Discuss

last person joined: 8 days ago 

Ask a question. Give advice. Share resources. Looking for software or hardware recommendations? Want to know how others are using text messaging in their work? Trying to find examples of IT policies? The NTEN Discuss forum is a great resource for all of the above and more! It's the general discussion list for the NTEN community, and folks all over the US (and the world) are sharing their questions, answers, and news about nonprofit technology.

What do you use for data-wrangling?

  • 1.  What do you use for data-wrangling?

    Posted Sep 10, 2019 16:43
    Hi all,

    I'm currently helping an organization transition from one AMS to another, and right now we're doing our data cleaning and organizing for transfer, mostly in Excel. Something tells me there must be an easier way to do this, and I'd love to hear your thoughts or recommendations on data wrangling tools (especially if they're free-ish or open source)

    Thanks all,

    ------------------------------
    Praan Misir
    Project Officer
    Ontario Library Association
    Toronto, ON
    ------------------------------
    2020 Nonprofit Technology Conference Logo  w/ Baltimore Skyline


  • 2.  RE: What do you use for data-wrangling?

    Posted Sep 11, 2019 11:27
    Hi Praan,

    Is the data transformation going to be one-time or on-going? If it's just one-time, and you don't have anyone on the team who knows SQL, then you might want to use a free-trial version of software like Trifacta, Talend, or Alteryx.  If this will be on-going, or your organization has other ETL needs (Extract, Transform, Load; aka data wrangling, transformation, and preparation) you could look into free relational database systems like PostgreSQL, or paid ones like Microsoft Azure SQL, or you could look into purchasing or subscribing to a product like the ones I mentioned above, which are more "business user"-focused rather than IT staff-focused.

    Albeit from vendors, here are relatively succinct summaries of data migration best practices from NetApp and Talend.

    Best,
    Lauren

    ------------------------------
    Lauren Feldman Hay (she/her/hers)
    Director of Informatics, Public Health Solutions
    New York, NY
    https://www.linkedin.com/in/lfeldmanhay/
    ------------------------------

    2020 Nonprofit Technology Conference Logo  w/ Baltimore Skyline


  • 3.  RE: What do you use for data-wrangling?

    Posted Sep 11, 2019 12:01
    Are you aware of Excel's Power Query? It's designed for data-cleansing.

    ------------------------------
    Oz du Soleil
    Milwaukie OR
    YouTube channel focused on Excel and Data-Cleansing
    https://www.youtube.com/c/OzduSoleilDATA

    Microsoft Excel MVP
    https://mvp.microsoft.com/en-us/PublicProfile/5001226?fullName=Oz%20du%20Soleil
    ------------------------------

    2020 Nonprofit Technology Conference Logo  w/ Baltimore Skyline


  • 4.  RE: What do you use for data-wrangling?

    Posted Sep 11, 2019 12:57
    I wonder what kind of cleansing and wrangling you're doing and how you're doing it.

    I've been cleansing data for almost 20 years--including helping people migrate CRMs. Excel has worked--especially with the introduction of Power Query about 5 years ago.

    ------------------------------
    Oz du Soleil
    Milwaukie OR
    YouTube channel focused on Excel and Data-Cleansing
    https://www.youtube.com/c/OzduSoleilDATA

    Microsoft Excel MVP
    https://mvp.microsoft.com/en-us/PublicProfile/5001226?fullName=Oz%20du%20Soleil
    ------------------------------

    2020 Nonprofit Technology Conference Logo  w/ Baltimore Skyline


  • 5.  RE: What do you use for data-wrangling?

    Posted Sep 11, 2019 13:48
    Thanks for the responses and recommendations Oz and Lauren. It's mostly error-correction, deduplication, and various odds and ends right now, and is really meant to be a one-time operation, so we'll be diving into those free trials :) . I'll also look into PowerQuery and see if we can make it work for us.

    Cheers,

    ------------------------------
    Praan Misir
    Project Officer
    Ontario Library Association
    Toronto, ON
    ------------------------------

    2020 Nonprofit Technology Conference Logo  w/ Baltimore Skyline