Within Python the best way to do file server scraping (or dumpster diving) is something like<\/p>\n
def find_OpPt_CDFs(obs,ver,targ_dir):\r\n file_pattern = re.compile('%s_d[ei]s\\d_engr_l1_sigthrsh_\\d{14}_v%s.cdf' % (obs,ver))\r\n \r\n print 'scraping in %s' % targ_dir\r\n \r\n filtered_files = []\r\n for root, dirs, files in os.walk(targ_dir):\r\n for file in files:\r\n if file_pattern.match(file):\r\n full_filename = (root+'\/'+file).replace('\\\\','\/')\r\n filtered_files.append(full_filename)\r\n return np.array(filtered_files).flatten()\r\n<\/pre>\nNote that the regex compile step uses:<\/p>\n
\r\n [ei] - either 'e' or 'i'\r\n \\d - digit\r\n \\d{14} - any 14 digits in a row\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"Within Python the best way to do file server scraping (or dumpster diving) is something like def find_OpPt_CDFs(obs,ver,targ_dir): file_pattern = re.compile('%s_d[ei]s\\d_engr_l1_sigthrsh_\\d{14}_v%s.cdf' % (obs,ver)) print 'scraping in %s' % targ_dir filtered_files = [] for root, dirs, files in os.walk(targ_dir): for file in files: if file_pattern.match(file): full_filename = (root+'\/'+file).replace('\\\\','\/') filtered_files.append(full_filename) return np.array(filtered_files).flatten() Note that the regex compile … Continue reading File Server Scraping<\/span>