Skip to content

transitmatters/data-ingestion

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

data-ingestion

lint test deploy

This is an app that can host all our data crunching jobs, especially those that use s3.

It uses Chalice, which makes it easy to add new functions that can be triggered on an interval or via rest API endpoint.

So far we have:

  • Store MBTA Alerts data daily.
  • Store number of trips with new trains on Orange and Red line daily.
  • Store Bluebikes station status data every 5 min.
  • Store ridership data
  • Process and store speed restrictions

To add a new lambda function, put the methods you need in a new file in chalicelib/. Then add your trigger in app.py. Then, add a policy.json file for any permissions you might need in .chalice/, updating .chalice/config.json accordingly. Lastly, if you need secret environment variables, update resources.json and make sure they are passed with --parameter-override in the cloudformation deploy step of deploy.sh. (Non-secret envvars can be added to config.json directly.)

About

Crontab for data ingestion/processing on AWS Lambda

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published