Tag: tech notes

Raspberry Pi 3B setup

install NOOBS from https://www.raspberrypi.org/downloads/noobs/ v 1.8.0 released 2016-02-29 Get online using wifi dongle and QMUL Guest login (fuck eduroam for now) >sudo apt-get update >sudo apt-get install mysql-server >sudo apt-get upgrade >sudo pip install PyMySQL Changed mind and went for MariaDB as that is what I use on PEW >sudo apt-get install mariadb-server As MySQL… Continue reading Raspberry Pi 3B setup

Intro to High Performance Computing (HPC) course

Taught by Lukas in eecs IT (25 nov 2015) http://www.eecs.qmul.ac.uk/~lukas/downloads/IntroductionToHPC.pdf Apocrita MPI – message passing interface Use Screen for qlogin sessions, incase your SSH connection goes, then your process will continue. Resource Allocation slide Core = slot in SGE smp (this gives you slots on the same machine will not spread over nodes) if you… Continue reading Intro to High Performance Computing (HPC) course

1st test – NMR tracking in lab (local version)

Today the iMac was deployed to track colony Omega (20 NMRs) using 7 sensors. The data is being stored on a local MySQL db. deployed code on github (misslake) estimated data logging (as determined with lights on): 24 rows of non-blank data per 30 seconds scales up to around 70,000 per day if NMR movement… Continue reading 1st test – NMR tracking in lab (local version)

Local iMac db setup

status at 20151117: Python script ok. Failing at the MySQL commit as the primary key stuff is not letting the data into the table. To do: add data to the sensor table. check PKs and FKs. check them again. Look at using python to check for duplicates instead of doing it this way.

Setting up iMac (OSXYosemite 10.10.5)

$ sudo easy_install pip $ sudo pip install -U setuptools $ sudo pip install pyserial install MariaDB as that is what is installed on pew == make sure Xcode is uptodate or install it $ xcode-select –install install homebrew $ ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)” Check Homebrew $ brew doctor Update Homebrew $ brew update… Continue reading Setting up iMac (OSXYosemite 10.10.5)

Rubular: regular expression editor and tester

Rubular is a Ruby-based regular expression editor and tester. It’s a handy way to test regular expressions as you write them. Rubular is an especially good fit for Ruby and Rails developers, since it uses Ruby on the server to evaluate regexes, but should also be useful for those working in other programming languages and… Continue reading Rubular: regular expression editor and tester

Soft robots

Hardware: Calibration button for hardware – bumper switch Software: Arduino Speed – can change the freq, by not using PWM. accelstepper – library to control acceleration. 3 motors ok with arduino – more motors might be tricky if added due to the switching speed. Might be best to have an arduino per motor, or per… Continue reading Soft robots

sepans Cheats – sepans

A cheat sheet containing scripts that I normally use (shell, sql, python .etc) Source: sepans Cheats – sepans contains some nice text sorting reg ex bits

Making the work survive the long-term / error handling

Notes: Long form work needs to have plans to keep it running (ref Ken Goldberg’s MementoMori, and LongPlayer and Walter de Maria https://en.wikipedia.org/wiki/New_York_Earth_Room) New NMRs and new tags – add system to pick up new tag and log it [detect if tag is in tag db, of not add it. check Primary Key status as… Continue reading Making the work survive the long-term / error handling

Python tips

Creating the code on Mac OS, to switch to Pi later ensure pip install and upgraded $ sudo pip install -U setuptools $ sudo pip install pyserial $ brew install mysql-connector-c ( on Mac OSX you need the mysql connection lib, will be installed if XCode installed apparently, see http://stackoverflow.com/questions/16182294/installing-mysql-python-on-mac-os-x) $ sudo pip install MySQL-python… Continue reading Python tips

Raspberry Pi set-up (old)

On mac, Download NOOBS from https://www.raspberrypi.org/downloads/noobs/ Unzip Format the microSD using SDFormatter software (I’ve used a 32GB card as I want to be able to backup the NMR data locally, just-in-case. This could be done to a USB stick too I guess). Copy files from root of NOOBS to freshly formatted SD card. Put card… Continue reading Raspberry Pi set-up (old)

Firewalld / iptables

on pew, i removed the iptables and engaged firewalld. These are firewalls that protect the server from unwanted incoming connections. Check status (you cannot run iptables AND firewalld, but firewalld is recommended now to usurp iptables) $ systemctl status iptables Disable and mask iptables $ sudo systemctl stop iptables $ sudo systemctl mask iptables Enable… Continue reading Firewalld / iptables

MySQL TIPS

for pew: sudo mysql (to give permissions) create database ; (only needs setting up once) use ; (before you start working on a db, make sure you are using the right one) create table …etc see below. show tables; describe ; DB: nmr TABLE: datatest CREATE TABLE datatest (time TIMESTAMP, recID TINYINT(10), avidID VARCHAR(16)); Check… Continue reading MySQL TIPS

Build your own Data Cloud – QMUL

UPLOAD SLIDES

MapReduce / Hadoop on RaspberryPi

If your data is processed once, maybe not keep it in the HFDS as it is slow(?) check. HDFS (Hadoop Distributed File SYstem).

Name Nodes store the metadata
Data Nodes store data (nodes are replicated a number of times (3 usually)

In nano hadoop-2.6.0/etc/hadoop/hadoop-env.sh (configuration file)
Edit the java implementation to use.
export JAVA_HOME=${JAVA_HOME}
to
export JAVA_HOME=”/home/pi/ejdk1.8.0_33/linux_armv6_vfp_hflt/jre/” [or equiv]

edit core-site.xml

fs.defaultFS
hdfs://pi-0:9000 <--- this is the master node. you have to place the tag inside the

edit hdfs-site.xml
dfs.replication
1 dfs.namenode.name.dir
/home/pi/had-hdfs/ dfs.datanode.data.dir
/home/pi/had-hdfs/

edit mapred-site.xml <--- may already be correctly edited mapreduce.framework.name
yarn

Yarn
– hadoop-2.6.0/etc/hadoop/yarn-site.xml

yarn.nodemanager.aux-services mapreduce_shuffle yarn.resourcemanager.hostname
localhost <---- localhost uses localhost as master. for NME ask ITS for the master IP on their Hadoop set-up  You need to start Hadoop process form the Master node... You need to start Yarn process form the Master node... [so both of these will be running at QMUL already] This was a SHIT workshop. to unzip tars in command line: tar xzvf [filename]

Installing software on the RPi 2 for testing the receivers:

Install node and node-red
$ curl -sL https://deb.nodesource.com/setup | sudo bash –

$ sudo apt-get install -y build-essential python-dev python-rpi.gpio nodejs

$ sudo npm install -g –unsafe-perm node-red

src [http://nodered.org/docs/hardware/raspberrypi.html]

Mainly using node-red in the first instance to test the serial port stuff as it’s so quick and easy. Will end up using python for the final version.

Working out power for all 7 readers:
Tested in the lab with a variable power supply.
each reader uses about 300mAh at 9v (they are usually powered by a 9v alkaline battery)
For 7 I need a 3A 9V plug-in power supply, and can then wire they all together into one source.
Ordered one plus a load of speaker cable to extend the power of each reader.

Data Storage at QMUL – meeting with Tim Kay.
Agreed that MySQL will probably be adequate for my needs. We calculated that the basic data (ID, scannerID, timestamp) would total about 10 Gb over 10 years. The data can be hosted on EECS as the whole lot will eventually be centralised and stored at IBM, so SBCS will have access in the future.
I can serve webpages for the front end, and will be able to use QMUL authentication for a web form that can be used to store the NMR measurements and so on, also accessible by Chris and his team.
I can have a separate domain name, but this may expire – would be better to have something that qmul will keep going … I like The Naked Mole Rat Diaries… and could buy a ten year domain, but need to think about this further.
Also should be fine to run a live stream of the data through an API (I will need this for my work anyway).
Regarding data privacy we think there is no issue as no personal (human) details stored.
Consider having a back-up / mirror of the data too (i.e. Amazon Cloud if they can give a free academic license? and can they provide a real-time feed?)

Powering the receivers – Ho suggests we wire them together and use a single power block that gives us 9v. We’re going to rig a few up and see what power they require on continuous read in the lab. Need to check that the current will be enough for all of them.
Ho also has an old UPS that I can have for the project – it needs new batteries, but that should be a cheaper option than getting a new one. Need to check if it is noisy for the NMRs.

Ordered the serial hub, power surge extension with USB and other bits. The hub will power the RPi, or it can be plugged directly into the power extension. This might be best. Also gives me some 5v ports to play with for any additional sensors.

Software will be written in Python, need to look into the local DB i.e.: https://oss.oetiker.ch/rrdtool/

NMR colony 800 map 20150519v1

OpenSensors.io training

IoT with OpenSensors – 28 April 2015 IoT deep dive with OpenSensors https://opensensors.io/ Node-RED – plumbing system for iot. nodered.org (opensource but developed at IBM Emerging Technology) @BorisAdryan node.js based webservice on various platfroms – Rpi, mac, cloud… website with GUI large library flows shared as JSON strings http://flows.nodered.org Node.js is a prerequirement, then run… Continue reading OpenSensors.io training

IoT deep dive with OpenSensors https opensensors io…

IoT deep dive with OpenSensors https://opensensors.io/

Node-RED – plumbing system for iot. nodered.org
@BorisAdryan

node.js based webservice on various platfroms – Rpi, mac, cloud…

  • website with GUI
  • large library
  • flows shared as JSON strings