for pew: sudo mysql (to give permissions) create database ; (only needs setting up once) use ; (before you start working on a db, make sure you are using the right one) create table …etc see below. show tables; describe ; DB: nmr TABLE: datatest CREATE TABLE datatest (time TIMESTAMP, recID TINYINT(10), avidID VARCHAR(16)); Check… Continue reading MySQL TIPS
Author: julie
Tools and resources for learning data analysis, visualization, social media
list of BEST tools and resources for learning data analysis, visualization, and working with social media data – Google Docs. from Lev Manovich
Terminal TIPS (linux)
This is a mix of Mac command line, RHEL, and Raspian commands — not all work on all platforms. $ sudo find / -name XXX [Find things] $ sudo -i [elevate your self to super user] $ cd / [change to root directory] $ cd ~ [change to home directory] $ ls -a [list everything… Continue reading Terminal TIPS (linux)
DataCite
We want to help make data more accessible and more useful; our purpose is to develop and support methods to locate, identify and cite data and other research objects. Specifically, we develop and support the standards behind persistent identifiers for data, and our members assign them. via What do we do? | DataCite.
re3data – schema
Schema for the Description of Research Data Repositories Version 2.2 December 2014 doi: http://doi.org/10.2312/re3.006 http://gfzpublic.gfz-potsdam.de/pubman/item/escidoc:758898:3/component/escidoc:775891/re3data_schema_v2-2_public_final-2014-12-03.pdf Research data are valuable and ubiquitous. The permanent access to research data is a challenge for all stakeholders in the scientific community. The long-term preservation and the principle of open access to research data offer broad opportunities for the scientific… Continue reading re3data – schema
Naming:
Nemacode – serverside
Critter – data vis / public facing
Today:
upgrade node-red to 0.10.10
start Python build, set up GIT on Pi
Soft Robotics – journal
Soft Robotics.
Seeing Data – research project
Seeing Data. Seeing Data is a research project which aims to understand how people make sense of data visualisations like the examples below. This website explains what we have been doing on Seeing Data, what we have found out, and how non-experts can develop their ability to make sense of data visualisations. The ‘Understanding Data Visualisations‘ section of the… Continue reading Seeing Data – research project
100 open source Big Data architecture papers
ralf baecker – portfolio
ralf baecker. physical constructions – review for data usage etc…
Ian Ingram | Animal Machines
Ian Ingram | Machines. Really relevant work for the NMR project intro. Go through further and explore use of data, anthropomorphisation, kinetic motion
art and sci in the post-biological age
http://videodoc.ncca-kaliningrad.ru/participants/iv-body-as-technology/ good repository for artworks (inc The Lake)
Algorithmic governmentality and the end(s) of critique–Antoinette Rouvroy
Society of the Query | Algorithmic governmentality and the end(s) of critique–Antoinette Rouvroy.
Quote: “knowledge is not produced about the world anymore, but form the world[..]” (i haven’t found quote yet – this was at Intrefaces seminar slides by Christian Ulrik Anderson.
Milestones in the History of Thematic Cartography, Statistical Graphics, and Data Visualization
Milestones in the History of Thematic Cartography, Statistical Graphics, and Data Visualization.
List of Physical Visualizations
Active physical visualization | List of Physical Visualizations.
Noumenon
The noumenon (/ˈnɒuːmɨnɒn/) is a posited object or event that is known (if at all) without the use of the senses.[1] The term is generally used in contrast with, or in relation to “phenomenon”, which refers to anything that appears to, or is an object of, the senses. In Platonic philosophy, the noumenal realm was… Continue reading Noumenon
ACM Computing Classification System, 2012 Revision
http://delivery.acm.org/10.1145/2380000/2371137/ACMCCSTaxonomy.html?ip=138.37.32.43&id=2371137&acc=OPEN&key=BF07A2EE685417C5%2E53A024A4F41048F4%2E4D4702B0C3E38B35%2E6D218144511F3437&CFID=683826054&CFTOKEN=51013308&__acm__=1434451434_4215bafbd152f8c206a318a68f6020fd
C4DM PhD Theses
Choosing R or Python for data analysis? An infographic
52 emails sent to people for my Data Vocab pilot survey.
From Paint to Pixels – data art
How Data Became a New Medium for Artists – The Atlantic.
Some good data artists referenced here.
Build your own Data Cloud – QMUL
UPLOAD SLIDES
MapReduce / Hadoop on RaspberryPi
If your data is processed once, maybe not keep it in the HFDS as it is slow(?) check. HDFS (Hadoop Distributed File SYstem).
Name Nodes store the metadata
Data Nodes store data (nodes are replicated a number of times (3 usually)
In nano hadoop-2.6.0/etc/hadoop/hadoop-env.sh (configuration file)
Edit the java implementation to use.
export JAVA_HOME=${JAVA_HOME}
to
export JAVA_HOME=”/home/pi/ejdk1.8.0_33/linux_armv6_vfp_hflt/jre/” [or equiv]
edit core-site.xml
edit hdfs-site.xml
edit mapred-site.xml <--- may already be correctly edited
Yarn
– hadoop-2.6.0/etc/hadoop/yarn-site.xml
Installing software on the RPi 2 for testing the receivers:
Install node and node-red
$ curl -sL https://deb.nodesource.com/setup | sudo bash –
$ sudo apt-get install -y build-essential python-dev python-rpi.gpio nodejs
$ sudo npm install -g –unsafe-perm node-red
src [http://nodered.org/docs/hardware/raspberrypi.html]
Mainly using node-red in the first instance to test the serial port stuff as it’s so quick and easy. Will end up using python for the final version.
Working out power for all 7 readers:
Tested in the lab with a variable power supply.
each reader uses about 300mAh at 9v (they are usually powered by a 9v alkaline battery)
For 7 I need a 3A 9V plug-in power supply, and can then wire they all together into one source.
Ordered one plus a load of speaker cable to extend the power of each reader.
Big Data and Humanitarian Response
Digital Humanitarians HOW BIG DATA IS CHANGING THE FACE OF HUMANITARIAN RESPONSE via Book | iRevolutions.
IEEE VISAP – conference
IEEE VIS 2015 Arts Program. Paper for the data vocabulary Artwork submission – We Need Us
Train timetables – working and active
The train timetables exist in two forms – working (set about every 6 months) and active (updated in real-time about an hour ahead). This site is a good example of where both data sets are used and accessible. Realtime Trains | 1L47 1550 London Waterloo to Gillingham (Dorset).
rotating plinth for objects
10cm mains/battery turntable. Could use these as an added component to the system – data driven turning linked to a behaviour
Hardware set-up today
Download Raspian
Install raspian on the RPi
Install wifi dongle
Problems getting onto Eduroam.
Will need to get the Pi on the network with a fixed IP
Orange Data Mining
Orange Data Mining. Tool for visualising diff data mining techniques
Tutorial Learn Python in 10 minutes Stavros’ Stuff…
William Ross Ashby: Intelligence Amplification
About selection and problem solving. How making a good choice is how we progress in all things. What happens when we have no choice? What happens when we make a ting choice and that is feeding into a bigger choice (voting, zooniverse, etc)? With some of these things we don’t know the end choices, we… Continue reading William Ross Ashby: Intelligence Amplification
Man-Computer Symbiosis paper by Licklider
“Man-Computer Symbiosis” is a key speculative paper published in 1960 by psychologist/computer scientist J.C.R. Licklider, which envisions that mutually-interdependent, “living together”, tightly-coupled human brains and computing machines would prove to complement each other’s strengths to a high degree: Man-computer symbiosis is a subclass of man-machine systems. There are many man-machine systems. At present, however, there… Continue reading Man-Computer Symbiosis paper by Licklider
Data Storage at QMUL – meeting with Tim Kay.
Agreed that MySQL will probably be adequate for my needs. We calculated that the basic data (ID, scannerID, timestamp) would total about 10 Gb over 10 years. The data can be hosted on EECS as the whole lot will eventually be centralised and stored at IBM, so SBCS will have access in the future.
I can serve webpages for the front end, and will be able to use QMUL authentication for a web form that can be used to store the NMR measurements and so on, also accessible by Chris and his team.
I can have a separate domain name, but this may expire – would be better to have something that qmul will keep going … I like The Naked Mole Rat Diaries… and could buy a ten year domain, but need to think about this further.
Also should be fine to run a live stream of the data through an API (I will need this for my work anyway).
Regarding data privacy we think there is no issue as no personal (human) details stored.
Consider having a back-up / mirror of the data too (i.e. Amazon Cloud if they can give a free academic license? and can they provide a real-time feed?)
Powering the receivers – Ho suggests we wire them together and use a single power block that gives us 9v. We’re going to rig a few up and see what power they require on continuous read in the lab. Need to check that the current will be enough for all of them.
Ho also has an old UPS that I can have for the project – it needs new batteries, but that should be a cheaper option than getting a new one. Need to check if it is noisy for the NMRs.
Ordered the serial hub, power surge extension with USB and other bits. The hub will power the RPi, or it can be plugged directly into the power extension. This might be best. Also gives me some 5v ports to play with for any additional sensors.
Software will be written in Python, need to look into the local DB i.e.: https://oss.oetiker.ch/rrdtool/
EPSRC research data principles
Research data: https://www.epsrc.ac.uk/about/standards/researchdata/ “The expectations arise from seven core principles which align with the core RCUK principles on data sharing. Two of the principles are of particular importance: firstly, that publicly funded research data should generally be made as widely and freely available as possible in a timely and responsible manner; and, secondly, that the… Continue reading EPSRC research data principles
Friday 8th May – tested the MiniTracker II [https://www.avidid.com/products/minitracker-i but with an RS232 cable] device with the NMR colony and received adequate readings from nearly all critters that passed through the tube.
The device does need resetting periodically – testing the timeout on this there appears to be no logical pattern, so as it is a low overhead I’ll send a
How to test?
- Establish the mean average frequency between readings
- Look for a number of segments of time when no data is coming from the device, that is above this average frequency.
Last week a total of six trackers were ordered and paid for. There is a possibility that we need an extra one if we want to establish time spent within the tubes and within each housing element.
PLOS ONE: A Distinct Role of the Queen in Coordinated Workload and Soil Distribution in Eusocial Naked Mole-Rats
PLOS ONE: A Distinct Role of the Queen in Coordinated Workload and Soil Distribution in Eusocial Naked Mole-Rats.
LAURIE FRICK | Walking, Eating, Sleeping
LAURIE FRICK | Walking, Eating, Sleeping. “Numbers are abstract concepts but we recognize pattern intuitively. I’m experimenting with wall size patterns that anticipate the condition of our daily-selves. Very soon walls and spaces we occupy will be filled with easy to decode patterns – a visual record of how we feel, stress level, mood, bio-function… Continue reading LAURIE FRICK | Walking, Eating, Sleeping
Degenerate hearing and sound localization in naked mole rats (Heterocephalus glaber), with an overview of central auditory structures. – PubMed – NCBI
Molecular evolution of the hyaluronan synthase 2 gene in mammals: implications for adaptations to the subterranean niche and cancer resistance | Biology Letters
Molecular evolution of the hyaluronan synthase 2 gene in mammals: implications for adaptations to the subterranean niche and cancer resistance | Biology Letters.
COUNTING IN EIGHT, MOVING BY COLOR – Program – KW Institute for Contemporary Art
COUNTING IN EIGHT, MOVING BY COLOR – Program – KW Institute for Contemporary Art.
OpenSensors.io training
IoT with OpenSensors – 28 April 2015 IoT deep dive with OpenSensors https://opensensors.io/ Node-RED – plumbing system for iot. nodered.org (opensource but developed at IBM Emerging Technology) @BorisAdryan node.js based webservice on various platfroms – Rpi, mac, cloud… website with GUI large library flows shared as JSON strings http://flows.nodered.org Node.js is a prerequirement, then run… Continue reading OpenSensors.io training
IoT deep dive with OpenSensors https opensensors io…
IoT deep dive with OpenSensors https://opensensors.io/
Node-RED – plumbing system for iot. nodered.org
@BorisAdryan
node.js based webservice on various platfroms – Rpi, mac, cloud…
- website with GUI
- large library
- flows shared as JSON strings
This week:
RFID system. New papers and insight from Susan Imberman and Dan McCloskey (CUNY) about their RFID tracking systems. They are willing to share the data they have, in real-time too when it’s ready.
TO DO: read papers carefully and ascertain what kind of analysis they are using and how relevant, if at all, to us.
Contacted Pet Detect who have a universal scanner that detects AVID tags. They are quoting for a system.
Plan B is to get Mike Harrison to build it. I am happy to just pay for this and get it installed, would rather focus on the software framework, which needs to be reusable and future proof, initial specifications developing here: https://docs.google.com/document/d/1mgJJ2LX2fCUtRphFdgwWl2YxhFc5v-MTgbadHfOahGE/edit
MATERIALS:
Skylar Tibbets et al – paper on active 3D printed materials. Thoroughly assessed this and am exploring how this technique can be used to activate the objects, would be best if not water activated but electrically conductive / heat responsive. Heat response material probably faster than the water expansion, and much more precision to target small areas.
In the paper, the author talks about self-evolving structures, which gives too much importance to the relatively simple motion driven by a state change. The object moves to a position and then returns to its start point. Self-transformation (used on page 5) is also not strictly true as the material changes in response to an environmental change, it doesn’t come from within itself. Anyway.
How can I combine a data-driven response with an object that has some innate movement? Does the data become a trigger, a time of activation? or permission for the thing to move? Would it be the ‘life’ in the object – i.e. without a data stream this object is inert (dead) even with it’s potential. Could the data affect the environmental conditions that the objects exist in as well as sending/modulating electrical signals?
The objects fabricated in these experiments are multi-materials embedded in a 3D structure. One material acts as a rigid core and the expanding (state-changing) material is positioned in such a way that when it expands it deforms the rigid sections. They refer to this as a Digital Material [http://www.stratasys.com/materials/polyjet/digital-materials] – a material that can be adjusted digitally (in terms of physical structure).
INTELLIGENT ENVIRONMENTS (lecture notes)
“The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.”
— Mark Weiser’s “The Computer for the Twenty-First Century” (Scientific American, 1991, pp. 66–75)
Intelligent Environments Manifesto – ref Augusto, Callaghan 2013 — in manifesto – one point says “to work under the principal that the user is in command” – WHY? Do we not need to relinquish control? How can we possibly command of something that we don’t know about? Yes these sensors can oversimplify reality but they bring to our attention more that we would OVERTLY think about usually… this is more complex?
ETHICAL ISSUES in (PRODUCT) DESIGN
NMR – what are the ethical issues to consider – can an object move in a subliminal way that make you feel awkward or at ease or stressed? How can these behaviours be recognised? Do we need to control for this if we are assuming the object is just communicating for it’s source. How far do we go in censoring the data? What are the ethical issues of data manipulation from a real-time stream of life data?
Body Language of Objects:
Great chat with Geraint about positioning the thesis around this title and concept.
Found some refs, but these both looking at different things. I’m interested in the actual body language of the object via data-driven movement:
http://www.chrisnatt.com/project/bodylanguage/
https://books.google.co.uk/books?id=iQnfJHjcVQ8C&pg=PA56&lpg=PA56&dq=%22body+language+of+objects%22&source=bl&ots=rnfcGKr0gJ&sig=h0qW13Mmh-LiO-8ufGDDsdOGVgc&hl=en&sa=X&ei=XwPhVMGwFIrfaK71gHg&ved=0CCwQ6AEwAg#v=onepage&q=%22body%20language%20of%20objects%22&f=false
GS: Cortexual not contextual.
Stats assignment 01 (mostly fiction):
Julie_Freeman_POLM036_20150213
this helps with choice of variable and tests: http://www.statsdirect.com/help/default.htm#methods/relationship_two_variables_measured_on_same_group.htm
My notes:
The null hypothesis is that there is NO association between the two variables. (i.e. women are not more likely to vote left than men)
We want to measure the significance of the association between varX and varY – do this using chi-squared test).
The closer to 1 means the LESS significant the relationship. Use for for category variables — i,e, gender vs the scale
In SPSS, the column marked Asymp. Sig. (2-sided) is the P value
P (probability) value breakdown:
P > 0.10 No evidence against the null hypothesis. The data appear to be consistent with the null hypothesis.
0.05 < P < 0.10 Weak evidence against the null hypothesis in favor of the alternative.
0.01 < P < 0.05 Moderate evidence against the null hypothesis in favor of the alternative.
0.001 < P < 0.01 Strong evidence against the null hypothesis in favor of the alternative.
P < 0.001 Very strong evidence against the null hypothesis in favor of the alternative.
We want to measure the strength and direction of the relationship using correlation coefficients
direction can be positive or negative (tendency)
strength is (i think ) the outcome of the Pearson or Spearman or other test:
- Pearson’s R – interval (scale) variables [use this for age vs L-R scale, do not use with gender]
- Spearman’s Rank – ordinal (ordered) variables
correlations: -1, 0, +1 (towards either end of scale (closer to 1/-1) = stronger relationship, signs indicate the direction NOT the strength)
Discovered paper of RFID tracking NMR, asked them to help explain the tech set-up. Paper not actually very insightful.
Kinetic Objects:
Think about how to collaborate with MIT on the programmable materials work – data-driving the materials from NMRs. With Skylar Tibbets
http://www.selfassemblylab.net/ProgrammableMaterials.php
http://www.nature.com/srep/2014/141211/srep07422/full/srep07422.html
Statistics:
Reading books on stats methods etc for SPSS
NMR scanning:
when the data finally comes in, check this paper Social network analysis: A methodological introduction by Butts.
Ontologies:
Linked Open Vocabularies – http://lov.okfn.org/dataset/lov/ a collection of ontologies about all sorts. Godo reference point for linked data bits. The ‘thing’ that I develop needs to be linked and in a correct format (RDF?) so it is machine as well as human usable.
http://www.w3.org/ns/ma-ont is a media ontology but does not cover the data as a material description that I am after.
The dublin core metadata terms has a great range of terms to be used: http://dublincore.org/documents/2012/06/14/dcmi-terms/ I think that the data art set should use terms from this but in an easy to use list with vocabularies. This could be in the same format as the Getty stuff perhaps.
The key issue is that there are many many ontologies, taxonomies and vocabularies to use, and as data sits behind all of the areas at some point, it is hard to choose a labelling standard. The data art model is a suggestion for ensuring that some basic data properties are captured and noted at the delivery of the work.
Do I make a small ontology just specifically for use with Data as an Art Material? Or will a simple table suffice? Do the relationships matter? And how would an artist then use the ontology? How to get the artists and the cataloguer to use the same things – does it matter? is it the artists job, or would the curator/classifier do this?


