Nonprofits and Data

last person joined: 8 days ago 

This group is for those interested in learning and sharing about all things data-related for nonprofits. The Nonprofits and Data group is for people using data to serve a mission, either directly or by improving nonprofits and the nonprofit sector. That includes everything from collecting data and managing databases to analytics, data visualization and data mining. Here are some examples of topics we discuss: using data to improve organizational effectiveness, measuring impact, using data for storytelling, tools for data management and analysis, figuring out the “right” data to collect, and learning skills to help us use data better.

Warehouses, pipelines, and bots, oh my!

  • 1.  Warehouses, pipelines, and bots, oh my!

    Posted 9 days ago
    Hi everyone,

    I'm looking into beefing up the usage of the data warehouse we use at our organization, and I'm wondering if anyone has found any good resources on creating/maintaining data pipelines and using bots to automate data extractions.

    My main concern is that many of the databases that we're mandated to use don't have APIs or automated export options. At this year's NTC, I attended a session on using bots to automate the ETL process, but I'm not sure if it's possible to do without more 'connectable' systems. Does anyone have experience with automating extracts from these types of systems? Or resources for how to go about setting up a structure for automating this process?

    Many thanks!


    Charles Riebeling
    Assistant Director of Learning and Evaluation
    Carlos Rosario International Public Charter School
    Washington, DC

  • 2.  RE: Warehouses, pipelines, and bots, oh my!

    Posted 8 days ago
    Hi Charles,

    When the source system does not have a robust API or automated export options, we have done this kind of work using bots that actually login to the system and perform the manual tasks that it takes to trigger and export or in some cases, run a report and scrape it to get the data.   We then transform that .csv file to the appropriate format and upload it to an FTP server where another bot picks it up and imports to the warehouse.

    Where there is a will.




    Molly Kelly
    Vice President, Digital Solutions

  • 3.  RE: Warehouses, pipelines, and bots, oh my!

    Posted 8 days ago
    Hi Charles,

    There are lots of ways to skin a cat. The main concern I have with investing in automation of systems that don't have APIs or automated export options is whether the goal of automation is obscuring the bad fit you may have with the system. It may be better to upgrade the underlying systems and applications that you're using, and get into a platform and environment that's more suitable to you in lots of ways, including making it easier to move data around automatically.

    Isaac Shalev
    Stamford CT

  • 4.  RE: Warehouses, pipelines, and bots, oh my!

    Posted 8 days ago
    Hi Charles,

    Isaac makes an excellent point. If your system cannot interface with a modern data warehouse, then trying to force connectivity might be like putting lipstick on a pig. Alternatively, it might make more sense to consider modernizing the said infrastructure.

    I should also add that cost-wise, a lot of times, what initially appears as a quick, inexpensive fix uncovers some other gaps, and ultimately this turns out to be an expensive, yet half-baked option that becomes more expensive to re-architect.

    Medha Nanal
    Strategic Data/Database Consultant for Nonprofits (Fundraising, Operations, Programs)