Unless otherwise directed, my usual approach is to start with serialized execution and then look for opportunities to parallel-ize execution. In many cases, I achieve parallel execution with a package I call a Controller. I’ve seen frameworks that rely on Controller packages, which are merely parent packages with hard-coded and precedence-constrained Execute Package Tasks. This sounds brute-force-ish because it is, but there are advantages such as deterministic parallel execution and, believe it or not, a printable process work flow diagram.

Determinism is important and easy to overlook. To avoid locking in a true work pile, you need to either define precedence or block concurrence to avoid locking. Neither requirement is trivial; either adds orders of magnitude to the complexity of an execution framework.

These are some considerations when designing data integration automation.

:{>

The post Some Considerations When Using the SSIS Work Pile Pattern appeared first on Linchpin People, LLC.

SSISAndScripting from Andy Leonard on Vimeo.

The demonstration code is available here.

:{>

The post Scripting And SSIS appeared first on Linchpin People, LLC.

It hasn’t. At least not yet.

My doctor and I speak about this almost each time I visit with him. The corporation that operates his practice nowadays seems endlessly locked in cycles of changing billing and EHR systems in search of low-cost compliance and integration. They’ve (literally) spent millions of dollars and my doctor hates the interfaces forced upon him and his patients (well, one, at least) hates the complexity of the billing and patient records systems. Can’t these systems all just get along?

The result? Higher medical data management costs. I’ll give you one guesses who pays these costs.

Munro posits the following from his article:

By at least one estimate (here) there are now about 500 independent EHR vendors.  Out of that large group is a subset of about 400 with at least one customer that has applied for Federal stimulus dollars through the labyrinthine process of meaningful use attestation. That would suggest a “first-cut” of about 100 vendors who made some commitment around certification – but have no reported customers (at least to date). That’s a staggering number of single-purpose software vendors for any industry to support – even bloated healthcare. The simple fact is it can’t. While there have been a few high-profile cases of EHR vendors shutting down, this last week was the first high-profile example of a vendor that was effectively decertified by the Feds for both their “ambulatory” and their “inpatient” EHR products. From the HHS.gov website last Thursday:

“We and our certification bodies take complaints and our follow-up seriously. By revoking the certification of these EHR products, we are making sure that certified electronic health record products meet the requirements to protect patients and providers,” said Dr. Mostashari.“Because EHRMagic was unable to show that their EHR products met ONC’s certification requirements, their EHRs will no longer be certified under the ONC HIT Certification Program.”

You may ask yourself, well, how did we get here? This, folks, is a mess. What’s missing? Applied standards.

“But Andy, you’ve told us standards slow down development!”

And I stand by that statement; standards do slow down development…unless you’re building interfaces. And then standards become the means for decoupled snippets, functions, methods, applications, and even platforms to communicate with each other. In some cases, we simply cannot be productive without standards – like TCP/IP. What would happen if everyone coded their own version of internet traffic? If that was the case, very few of you would reading this post.

Yes, standards slow things down. And yes, they are necessary to insure base functionality. In my humble opinion, we have to get this right with healthcare data. We simply must. While we see similar issues of data management across many fields, medical data is too important to mess around with; it’s (often literally) life and death. And it is certainly a high cost.

More to Consider

Standards exist. Administering and certifying 400-500 vendor solutions is hard.

Part of the Solution

From the actions of the Department of Health and Human Services last week, one can ascertain HHS is taking steps to address the matter. But will all 400-500 companies voluntarily congeal their schemas? Possibly, but doubtful.

My experience delivering US state Medicaid ETL solutions informs me there will be a need for data integration – regardless of the existence of standards and in spite of certification. Why? Standards are not static. The idea of de facto standards emerges from the life cycle of software because software is organic. Even if everyone agreed on the same interpretation of rigid standards (and they won’t), versions 2.0 through n.n will – at a minimum – add fields to the schema. And with additional fields comes additional data.

Standards will be revised when enough product schemas adopt the de facto, and this will drive the need for yet more integration. Don’t take my word for it, examine the entropic history of ICD-9 and ICD-10 codes – the direction of progress is more data, not less.

Learn More

This is one reason we at Linchpin People are focusing on Medical Data Integration. The recording of our first (free!) webinar about Medical Data Integration with SSIS 2012 is available here. Kent Bradshaw and I continue the series tomorrow presenting Medical Data Integration with SSIS 2012, Part 2 in which we focus on loading Provider and Drug data.

I hope to see you there!

:{>

The post Little Data Remains Important in Healthcare IT appeared first on Linchpin People, LLC.

Register today!

:{>

The post Scripting and SSIS – A Free Webinar 8 May! appeared first on Linchpin People, LLC.

The recording of the first part of this series is available here.

:{>

The post Medical Data Integration with SSIS 2012, Part 2 – 1 May 2013 11:00 AM EDT appeared first on Linchpin People, LLC.

Linchpin People announces the availability of the SSIS Event Task for SSIS 2012 x64!

The Event Task simplifies adding messages to the SSIS log and raising events with custom messages at specific points during execution.

To configure, drag the Event Task onto the Control Flow:

Open the editor and set the EventName, EventDescription, EventCode, and SubComponent properties:

When the package executes in SSDT you can view the message on the Progress / Execution Results tab:

The task is priced at only $49.99 USD!

For more information, I created a demo screencast:

LPEventTask2 from Andy Leonard on Vimeo.

The post Announcing the Linchpin People SSIS Event Task appeared first on Linchpin People, LLC.

Kent and I had an AWESOME time presenting Medical Data Integration with SSIS 2012, Part 1! We hope you enjoy the video:

MedicalDataIntegrationWithSSIS2012 Part1 from Andy Leonard on Vimeo.

:{>

The post Medical Data Integration with SSIS 2012, Part 1 – Loading Claims Data appeared first on Linchpin People, LLC.

Espero que se unan a mí!

:{>

The post Presenting at SQL Saturday 189–Costa Rica! appeared first on Linchpin People, LLC.

Break it into Pieces

Did you hear about that person who climbed Mt. Everest in a single step? Me neither. Break that big project down into smaller steps… and then resist the urge to collect metrics about each step. (More later…)

Start Small

Once you have the big project broken into smaller chunks, start executing on the chunks!

Don’t Stop

Something has to give. Fine. Make the obstacles give up before you give up.

The Metric

If you start producing someone will show up and want to measure your work. You know, to help. They will be drawn to your practice of breaking down the solution into smaller chunks because this provides more things to measure.

You cannot divide art.

Well, you can, but it’s no longer art – it’s pieces formerly known as art. Measuring the steps is like taking scissors to the Mona Lisa.

There is only one metric. Like da Vinci’s famous painting, software is either finished or not.

Courage <> Absence of Fear

Courage is action in spite of fear. If you are waiting until you have no fear before moving forward, you will never move forward.

Conclusion

Do not ask, “What if I fail?” You are going to fail – plan on it. And then transcend the failure. Rinse, repeat; until you succeed. You can do anything you set your mind to accomplish.

The post “What if I fail?” appeared first on Linchpin People, LLC.

I like Red Gate’s SQL Source Control for SQL Server a lot. At Linchpin People, we are using SQL Source Control for several projects. It. just. works. It’s so cool to be able to store the database schema in the same source control engine as the other code (SSIS or .Net for us). We even use it with the Team Foundation Service which is free for teams of five or fewer users.

Barriers to entry are dropping like flies!

If you use Oracle and develop databases, I encourage you to participate in my little contest:

My Little Contest

To win, all you have to do is tell me about the cool stuff you’re going to do with your free SQL Source Control for Oracle license. Add a comment to this blog with your response. (Comments are moderated and will not appear right away.)

That’s it!

Winners will be announced in a few weeks.

:{>

The post Free Licenses: Red Gate SQL Source Control for Oracle! appeared first on Linchpin People, LLC.