Building Data Pipelines Connecting External Sources To Database

This article throws light upon how to build data pipelines connecting external sources to databases. Dropbase specializes in integrating data sent as email attachments (and more) with your data pipeline.

This article was originally published by Dropbase.

Data pipelines can have many sources. Many of these cases are either covered in this article about converting csvs to databases or this one about excel files. Some of these sources could be external to your organization. You could be requesting data from one of your sister companies on a weekly basis. Or you’ve outsourced data entry to another organization and constantly pass data around on emails. It can be frustrating to having to always remind your business partners about that monthly POS. The pain points are endless. We’re going to cover these problems in depth and show how Dropbase can solve them.

Regardless of which Dropbase feature listed below you use to solve your problem, we’re going to save you time and make it convenient to work with data. You no longer have to download that csv, re-clean it, and then finally upload to database. Both solutions eliminate the hassle of working with data the same way everyday. Use any of these features and all you’ll have to do is click approve. Let’s get into it.

Dropmail Helps You Unlock Data Sent Through Email—and Automates it as Well

Do you get a flat file mailed to you frequently where you download it, clean it the same way as last time, and then merge it with your data pipeline? Here’s how simple your email integration could be:

Quick Preview of Dropmail – Watch Video

In that video, a data sender mails over the data they usually do but to a special email address that automatically accepts the data, and starts a predefined cleaning routine before loading it to your database. All you have to do is click “Approve”.

Dropmail (and many other Dropbase features!) is ready for you to use if you’d like to sign up for a trial here.

Diving Deeper into Dropmail—How to Make Email Work for You

Here’s how to set up a Dropmail Request:

1. Head over to a Table you’d like to create a Dropmail for and click on Pipelines

2. Click Create New Pipeline and click on the Dropmail tab

3. Here, you can give your Dropmail a meaningful name. For example, if this is a monthly occurring sales file from Douglas Adams, you can call it “Monthly Sales from Douglas”. You can also select which pre-made cleaning steps you’d like to apply to it

4. Click on Create Dropmail and you now have an email address that anyone can use to send data to your pipeline!

A Few Reasons Why Dropmail is Amazing:

• The data sender can send multiple attachments in one go. These attachments go to the same pipeline and will be loaded in after you approve them.
• The sender doesn’t have to change their workflow. Instead of emailing you directly, they can just cc you and send it to the Dropmail instead. No programs to install, no accounts to make.
• You can pass the email to as many data senders as you like. If you get the same type of file from a bunch of senders, simply send them the same Dropmail. You’ll get all the notification right inside Dropbase and not have to worry managing data in your email inbox.

Who’s Dropmail Not For:

From above, we learned that Dropmail is for anyone who gets flat file data sent over email repeatedly and cleans them for their pipeline. The key point is that the data sender usually sends the data as an email attachment. What happens if your data source isn’t email but still sent by an external source? Here’s some examples:

• The data sender logs into their retail reporting software (e.g. Walmart Retail Link), downloads the file, and saves it in a shared folder that you have to navigate through.
• The emails you’ve been receving are getting larger and larger. They’re reaching the limits of attachment size (it’s 25MB for Gmail) and you might have to work with links. Nobody wants to do that.

In these cases, we need a different Dropbase feature to help us fully utilize our data. Introducing Dropzone.

Dropzone Gives Data Senders a Better Way to Upload Flat Files

Instead of sending data over email, a shared folder link buried in ancient folders, or something else, all the data sender has to do is drag and drop their data into Dropbase. This is how you set up a Dropzone request:

1. Just like for Dropmail, click the create new pipeline button in the Pipelines tab of an existing table.
2. Fill out the Dropzone name (e.g. what the data is: Weekly Store POS), data sender’s email, any cleaning steps, and click Create Dropzone.
3. Once the data sender uploads the data to Dropbase, you’ll get a notification and all you have to do is click Approve to load the data to your database.

Here’s what it looks like in Dropbase:

Dropzone Overview – Watch Video

Dropzone requires the data sender to make a Dropbase account but the data sender pays nothing. They don’t have to sign up for a trial account or be added to your Team on Dropbase. This makes organizing your data sources a lot more convenient.

In Conclusion

Both Dropzone and Dropmail help you save time and energy importing flat files from external sources. They make it easier to repeatedly work with the same kind of data, so you never clean the same data twice manually. Dropmail is best when you’re sending files over email that meet your email provider’s size limit. If you’re receiving email attachments larger than (usually) 25MB, then Dropzone lets your data sender upload the files—at no cost to them—and all you have to do is click Approve.