Showing posts with label flat. Show all posts
Showing posts with label flat. Show all posts

Friday, February 24, 2012

Create Flat File source programatically

I created a SSIS package, added Script task. created data flow task programatically, trying to add a flat file source component programatically. stuck at this point.

my goal is to add flat file source component to the data flow task and insert into a table in sql server using oledb destination component all programatically.

any help is appreciated. thanks.

You have to build a Flat File Connection Manager first and then reference it in your Flat File Source.

Have you looked at the BOL and previous posts in this forum for guidance?

Thanks.

|||

Iam creating the connection, I want know how I can read the flat file, and insert into a table. he is the code Iam using....

' Add the component to the dataFlow metadata collection

flatfileSource = dataFlowTask.ComponentMetaDataCollection.New()

' Set the common properties

flatfileSource.ComponentClassID = "DTSAdapter.FlatFileSource"

flatfileSource.Name = "FlatFileSource"

flatfileSource.Description = "Flat file source"

' Create an instance of the component

Dim inst As CManagedComponentWrapper = flatfileSource.Instantiate()

inst.ProvideComponentProperties()

' Associate the runtime ConnectionManager with the component

flatfileSource.RuntimeConnectionCollection(0).ConnectionManagerID _

= package.Connections("FlatFileConnection").ID

flatfileSource.RuntimeConnectionCollection(0).ConnectionManager _

= DtsConvert.ToConnectionManager90( _

package.Connections("FlatFileConnection"))

|||

You need to configure your connection manager by defining the file to use, column formats, and all additional properties before setting up your source.

There are also three more methods to call on your flat file source after the code you displayed:

AcquireConnections()

ReinitializeMetadata()

ReleaseConnections()

After that, you should hook up your source with the rest of a data flow and try to execute it.

HTH.

|||

sorry, Iam not explaining the problem correctly.

I do have those 3 lines of code. Iam lot where I have to loop through the columns or rows of my flat file. and how to map the columns to the destination table columns. any good example or reference please. thanks.

|||

Iam using the CreatePackage example in the samples. Iam able to create flat file source connection. not able to read the file.

Private Sub MapFlatFileDestinationColumns()

Dim wrp As CManagedComponentWrapper = flatfileDestination.Instantiate()

Dim vInput As IDTSVirtualInput90 = flatfileDestination.InputCollection(0).GetVirtualInput()

For Each vColumn As IDTSVirtualInputColumn90 In vInput.VirtualInputColumnCollection

wrp.SetUsageType(flatfileDestination.InputCollection(0).ID, vInput, vColumn.LineageID, DTSUsageType.UT_READONLY)

Next

' For each column in the input collection

' find the corresponding external metadata column.

Dim exCol As IDTSExternalMetadataColumn90

For Each col As IDTSInputColumn90 In flatfileDestination.InputCollection(0).InputColumnCollection

exCol = flatfileDestination.InputCollection(0).ExternalMetadataColumnCollection(col.Name)

wrp.MapInputColumn(flatfileDestination.InputCollection(0).ID, col.ID, exCol.ID)

Next

End Sub

trying use this code to map the columns, now my flat file connection is a source instead of destination. hope Iam explaining the problem.

thanks.

|||

External columns are automatically mapped to output columns when you call ReinitializeMetadata of your flat file source.

A good way for testing this would be to configure your package to some point (only a flat file source initially), save the package to a file and then open that file in the designer and inspect the metadata using the advanced component editor.

Thanks.

|||

Hi, i'm trying to 'emulate' a flat file source component using the script component in a data flow task. I need to be able to create a flat file connection and get the columns in the script (the columns change now and then, so i can't create a 'fixed' flatfile connection manager).

After getting the columns, i need to create output columns for the script component and to go through each row of the file and assign the column details.

I have derived column transformation after this source file step, do i need to make changes to the input / output columns of the derived column component in any way?

I also have an oledb destination, how can i generate an sql command to be used in the oledb destination? I know i can use sql from variable in an oledb source, but i can't find it in the oledb destination

I don't need real code examples, but i need some help on what objects need to be created, what functions need to be called etc... I'm very new to ssis programming.

Create Flat File source programatically

I created a SSIS package, added Script task. created data flow task programatically, trying to add a flat file source component programatically. stuck at this point.

my goal is to add flat file source component to the data flow task and insert into a table in sql server using oledb destination component all programatically.

any help is appreciated. thanks.

You have to build a Flat File Connection Manager first and then reference it in your Flat File Source.

Have you looked at the BOL and previous posts in this forum for guidance?

Thanks.

|||

Iam creating the connection, I want know how I can read the flat file, and insert into a table. he is the code Iam using....

' Add the component to the dataFlow metadata collection

flatfileSource = dataFlowTask.ComponentMetaDataCollection.New()

' Set the common properties

flatfileSource.ComponentClassID = "DTSAdapter.FlatFileSource"

flatfileSource.Name = "FlatFileSource"

flatfileSource.Description = "Flat file source"

' Create an instance of the component

Dim inst As CManagedComponentWrapper = flatfileSource.Instantiate()

inst.ProvideComponentProperties()

' Associate the runtime ConnectionManager with the component

flatfileSource.RuntimeConnectionCollection(0).ConnectionManagerID _

= package.Connections("FlatFileConnection").ID

flatfileSource.RuntimeConnectionCollection(0).ConnectionManager _

= DtsConvert.ToConnectionManager90( _

package.Connections("FlatFileConnection"))

|||

You need to configure your connection manager by defining the file to use, column formats, and all additional properties before setting up your source.

There are also three more methods to call on your flat file source after the code you displayed:

AcquireConnections()

ReinitializeMetadata()

ReleaseConnections()

After that, you should hook up your source with the rest of a data flow and try to execute it.

HTH.

|||

sorry, Iam not explaining the problem correctly.

I do have those 3 lines of code. Iam lot where I have to loop through the columns or rows of my flat file. and how to map the columns to the destination table columns. any good example or reference please. thanks.

|||

Iam using the CreatePackage example in the samples. Iam able to create flat file source connection. not able to read the file.

Private Sub MapFlatFileDestinationColumns()

Dim wrp As CManagedComponentWrapper = flatfileDestination.Instantiate()

Dim vInput As IDTSVirtualInput90 = flatfileDestination.InputCollection(0).GetVirtualInput()

For Each vColumn As IDTSVirtualInputColumn90 In vInput.VirtualInputColumnCollection

wrp.SetUsageType(flatfileDestination.InputCollection(0).ID, vInput, vColumn.LineageID, DTSUsageType.UT_READONLY)

Next

' For each column in the input collection

' find the corresponding external metadata column.

Dim exCol As IDTSExternalMetadataColumn90

For Each col As IDTSInputColumn90 In flatfileDestination.InputCollection(0).InputColumnCollection

exCol = flatfileDestination.InputCollection(0).ExternalMetadataColumnCollection(col.Name)

wrp.MapInputColumn(flatfileDestination.InputCollection(0).ID, col.ID, exCol.ID)

Next

End Sub

trying use this code to map the columns, now my flat file connection is a source instead of destination. hope Iam explaining the problem.

thanks.

|||

External columns are automatically mapped to output columns when you call ReinitializeMetadata of your flat file source.

A good way for testing this would be to configure your package to some point (only a flat file source initially), save the package to a file and then open that file in the designer and inspect the metadata using the advanced component editor.

Thanks.

|||

Hi, i'm trying to 'emulate' a flat file source component using the script component in a data flow task. I need to be able to create a flat file connection and get the columns in the script (the columns change now and then, so i can't create a 'fixed' flatfile connection manager).

After getting the columns, i need to create output columns for the script component and to go through each row of the file and assign the column details.

I have derived column transformation after this source file step, do i need to make changes to the input / output columns of the derived column component in any way?

I also have an oledb destination, how can i generate an sql command to be used in the oledb destination? I know i can use sql from variable in an oledb source, but i can't find it in the oledb destination

I don't need real code examples, but i need some help on what objects need to be created, what functions need to be called etc... I'm very new to ssis programming.

Sunday, February 19, 2012

Create Dimension Table from Fact Table!!

I have picked an exmple from this forum, to help me explain my current problem...

"I'm looking for a solution to import data from a flat file into an normalized data modell. To explain it a little simpler think about to following:

The Data Souce is a CSV-File with FirstName, LastName and Category. Sample data could be

Dirk; Bauer; sailing
Peter; Bauer; fishing
Marc; Bauer; reading

In my data modell I have defined the 2 tables "Person" and "Category":

Table "Person"
-
[PersonID] [int] IDENTITY(1,1) NOT NULL
[CategoryID] [int] NOT NULL
[FirstName] [nvarchar](50)
[LastName] [nvarchar](50)

Table "Category"
-
[CategoryID] [int] IDENTITY(1,1) NOT NULL
[CategoryName] [nvarchar](50)

Now I like to read my first row from the source and lookup a value for the CategoryID "sailing". As my data tables are empty right now, the lookup is not able to read a value for "sailing". Now I like to insert a new row in the table "Category" for the value "sailing" and receive the new "CategoryID" to insert my values in the table "Person" INCLUDING the new "CategoryID".

I think this is a normal way of reading data from a source and performing some lookups. In my "real world" scenario I have to lookup about 20 foreign keys before I'm able to insert the row read from the flat file source.

I really can't belief that this is a "special" case and I also can't belief that there is no easy and simple way to solve this with SSIS. Ok, the solution from Thomas is working but it is a very complex solution for this small problem. So, any help would be appreciated...

Thanks,
Dirk"

http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=74752&SiteID=1

Could someone help me creating the dimension table?

Thanks!!

It's very common to derive dimensions from source data - that is the purpose of a data warehouse after all.

What you need to do is perform the dimension lookup with your source data. For values not found (New dimensions), you can run the records down the error output into a Derived Column derivation to create your new dimension table complete with keys, then merge them back into the flow to update the dimension tables. Your methodology will determine how this is done.

|||How about seperate packages for dimensions and facts. The dimensions get built first from the source data so there will always be a lookup match when building the fact|||That will work as well. Always transform dimensions before facts.

Wes|||

wesd wrote:

That will work as well. Always transform dimensions before facts.

Wes

That's what I would do.

|||Please mark this thread as answered, the op has the information he needs. Email me if you need more information.|||

I will Try!!

Thanks!!!