Masked Scientific Arrays

Define a typical scientific array

The definition of the matrix A is designed to represent a typical scientific data set that varies by orders of magnitude. In addition, some of the data is invalid due to either no measurement () or there was a measurement glitch ().

For sake of completeness, the entire matrix is given by:

In [16]:
A = np.array([[1e-4,0,1e-5],[1e-3,5e-3,-1]])
print A
[[  1.00000000e-04   0.00000000e+00   1.00000000e-05]
 [  1.00000000e-03   5.00000000e-03  -1.00000000e+00]]

A simple matrix plot (using matplotlib's pcolormesh) doesn't show anything useful since all the 'good' numbers are orders of magnitude smaller than the bad. Note also that the matrix is 'plotted up' as the zeroth row corresponds to the lowest row of the plot

In [17]:
plt.pcolormesh(A)
Out[17]:
<matplotlib.collections.QuadMesh at 0x862fab0>

Of course, the way to bring out the data is by taking a logarithm - but the inclusion of the '0' and the '-1' are going to cause problems. Numpy gives two warnings (different ones) and it comes back with different values. The resulting matrix should look like

In [20]:
log_A = np.log10(A)
print log_A
[[-4.          -inf -5.     ]
 [-3.      -2.30103      nan]]
C:\Users\Conrad\Anaconda2\lib\site-packages\ipykernel\__main__.py:1: RuntimeWarning: divide by zero encountered in log10
  if __name__ == '__main__':
C:\Users\Conrad\Anaconda2\lib\site-packages\ipykernel\__main__.py:1: RuntimeWarning: invalid value encountered in log10
  if __name__ == '__main__':

This helps a bit, but unfortunately the 'bad' values put in a false color that says something it shouldn't.

In [21]:
plt.pcolormesh(log_A,vmin=-6,vmax=-3)
Out[21]:
<matplotlib.collections.QuadMesh at 0x870c4b0>

A much better alternative is to mark the bad points as invalid. Numpy's masked array does this for us. The correct invoccation is numpy.ma.masked_invalid, where ma is numpy's masked array package and masked_invalid is the function that takes care of $-\infty$ and $nan$. The proper array is now

In [22]:
masked_log_A = np.ma.masked_invalid(log_A)
print masked_log_A
[[-4.0 -- -5.0]
 [-3.0 -2.3010299956639813 --]]
In [23]:
plt.pcolormesh(masked_log_A,vmin=-6,vmax=-3)
Out[23]:
<matplotlib.collections.QuadMesh at 0x875cc90>

Smoothing Jupyter Startup

Within:

C:\Users\\.ipython\profile_default\startup

place whatever files you want executed with the following naming convention

00-global_imports.py
01-local_imports.py
02-global_magics.ipy

with contents like:

00-global_imports.py

#global imports
import matplotlib as mpl
mpl.use('agg')

import datetime          as dt
import matplotlib.cm     as cmap
import matplotlib.dates  as mdates
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker
import numpy             as np
import os 
import re
import scipy             as sp
import scipy.interpolate as interp
import scipy.optimize    as opt
import scipy.special     as special
import spacepy.pycdf     as pycdf
import sys
02-global_magics.ipy

#magic happens
%load_ext autoreload
%autoreload 2
%matplotlib inline

File Server Scraping

Within Python the best way to do file server scraping (or dumpster diving) is something like

def find_OpPt_CDFs(obs,ver,targ_dir):
    file_pattern = re.compile('%s_d[ei]s\d_engr_l1_sigthrsh_\d{14}_v%s.cdf' % (obs,ver))
        
    print 'scraping in %s' % targ_dir
 
    filtered_files = []
    for root, dirs, files in os.walk(targ_dir):
        for file in files:
            if file_pattern.match(file):
                full_filename = (root+'/'+file).replace('\\','/')
                filtered_files.append(full_filename)
    return np.array(filtered_files).flatten()

Note that the regex compile step uses:

    [ei] - either 'e' or 'i'
    \d   - digit
    \d{14} - any 14 digits in a row

Block Quotations

  • Common Cents

.myQuoteDiv {
background-color: #f5f5dc;
border-style: none;
border-left: 6px solid #d5d5bc;
width: 500px;
margin: 0px 0px 16px 50px;
padding: 4px 4px 4px 10px;
}

.myQuoteDiv  .myAttrib {
margin-top: 8px;
font-style: italic;
}

<div class = "myQuoteDiv">Chipotle’s questionable marketing tactics, and similar recent marketing moves by Whole Foods and Panera, may have brought out the knives of scientists and science minded journalists, but they were not hatched in a vacuum. Consumers are responding to the demonization of GMOs and our foods are tainted with ‘dangerous’ chemicals.
<div class = "myAttrib">- Jon Entine & Rebecca Randall</div>
</div>

Color schemes for the Columns

The color scheme for the highlight boxes for each of the columns are:

Under the Hood: <div style="background-color: #d5d5dc; border: solid 1px black;">

Common Cents: <div style="background-color: #f5f5dc; border: solid 1px black;">

Aristotle to Digital: <div style = "background-color:#ff6666; border: solid 1px black;">

About Comics: <div style="background-color: #dee7d2;border: solid 1px black">

Numerically solve an ODE in Maxima

As an example, solve the SHO numerically in Maxima

  1. load("dynamics")
  2. Specify only the RHS of . Note that dx_dt and dvx_dt are simply names of expressions
    • dx_dt         : vx
    • dvx_dt       : -omega^2 * x
    • my_odes  : [dx_dt,dvx_dt]
  3. Define the variable names
    • var_names : [x,vx]
  4. Specify initial conditions
    • x0    : 1
    • vx0 : 0
    • ICs : [x0,vx0]
  5. define the domain of time (independent variable name, span and step) over which the equations will be solved
    • t_var          : t
    • t_start       : 0
    • t_stop        : 10
    • t_step        : 0.1
    • t_domain : [t_var, t_start, t_stop, t_step]
  6. Issue the command to run
    • Z : rk(my_odes,var_names,ICs,t_domain)

Installing the LeatherDiary Theme

  1. Log into JustHost (not the WP admin on the client side but the admin on the server side)
  2. Locate the leatherdiary-premium directory under /public_html/wp-content/themes
  3. Locate the target subdomain (e.g. UnderTheHood)
  4. Copy the leatherdiary-premium directory whole from /public_html/wp-content/themes to /public_html/<subdomain>/wp-content/themes  (e.g. the final directory structure for UnderTheHood would look like public_html/UnderTheHood/wp-content/themes/leatherdiary-premium)
  5. Now log into as WP admin on the subdomain side
  6. Goto 'Appearance' on the left widget
  7. Pick 'Themes' from there and then activate Leather Diary Premium

Changing the Appearance of the WordPress Installation

1) Log into subdomain on the WordPress side as ‘Admin’ (using pre-selected username and password – see Making a Subdomain)
2) From the left widget select 'Settings' (note that 'Settings' is only visible for 'Admin')
3) Under 'Settings' select:

  • 'General' to change the Site Title and Tagline
  • 'Writing' to change how posts appear, are categorized, or converted from email
  • 'Reading' for how long posts stay active and how syndication and search engines work
  • 'Discussion' for how the site manages comments, links from other sites, etc.
  • 'Media' for how pictures are handled
  • 'Permalinks' for how the style of permalinks is set