Talend has several connectors to integrate SAP systems. However, this guide is no introduction to Talend’s SAP components. Instead, this guide helps to
- understand different alternatives to integrate SAP systems with Talend
- set up a local SAP system
- configure Talend Studio for using SAP components
- use Talend’s SAP wizard
- run a first Talend Job which connects to SAP
All further required information and example use cases for Talend’s SAP components should be available in the Talend component guide at www.help.talend.com. If that’s not the case, please create a JIRA Documentation ticket (https://jira.talendforge.org/browse/DOCT)!
Now let’s take a look at different alternatives for integration of SAP systems with Talend.
Alternatives for SAP Integration
Three protocols exist for communication between SAP and external programs:
- Dynamic Information and Action Gateway (DIAG): e.g. used by SAP GUI
- Remote Function Call (RFC): A function call with input and output parameters (like a Java interface)
- Hypertext Transfer Protocol (HTTP): internet standard
The following alternatives are available for integrating SAP systems using some of these protocols.
SAP supports the direct import of files (Call-Transaction-Program, Batch-Input, Direct Input). Files have to be in a specific format to be imported. Transformation and integration can be realized with Talend’s various File components such as tFileInputDelimited.
Remote Function Call is the proprietary SAP AG interface for communication between a SAP System and other SAP or third-party compatible system over TCP/IP or CPI-C connections. Remote function calls may be associated with SAP software and ABAP programming, and provide a way for an external program (written in languages such as PHP, ASP, Java, or C, C++) to use data returned from the server. Data transactions are not limited to getting data from the server, but can insert data into server records as well. SAP can act as the Client or Server in an RFC call.
A Remote Function Call (RFC) is the call or remote execution of a Remote Function Module in an external system. In the SAP system, these functions are provided by the RFC interface system. The RFC interface system enables function calls between two SAP systems, or between a SAP system and an external system.
tSAPInput and tSAPOutput are Talend’s components to use RFCs.
Business Application Programming Interface (BAPI)
A BAPI is an object-oriented view on most data and transactions of a SAP system (called “business objects”). Object types of the business objects are stored in the Business Object Repository (BOR).
BAPIs are always implemented as RFCs and therefore can be called the same way. Additionally, they have the following characteristics (compared to RFCs):
- Stable interface
- No view layer
- No exceptions, instead export parameter: “RETURN”
Most business objects offer the following standard BAPIs:
tSAPInput and tSAPOutput are Talend’s components to use BAPIs.
Application Link Enabling (ALE)
Application Link Enabling (ALE) is used for asynchronous messaging between different systems via “Intermediate Documents (IDoc)”. IDoc is a SAP document format for business transaction data transfers. It is used to realize distributed business processes.
IDoc is similar to XML in purpose, but differs in syntax. Both serve the purpose of data exchange and automation in computer systems, but the IDoc technology takes a different approach. While XML allows having some metadata about the document itself, an IDoc is obligated to have information at its header like its creator, creation time, etc. While XML has a tag-like tree structure containing data and meta-data, IDocs use a table with the data and meta-data. IDocs also have a session that explains all the processes which the document passed or will pass, allowing one to debug and trace the status of the document.
An IDoc consists of
- Control record (it contains the type of IDoc, port of the partner, release of SAP R/3 which produced the IDoc, etc.)
- Data records of different types. The number and type of segments is mostly fixed for each IDoc type, but there is some flexibility (for example an SD order can have any number of items).
- Status records containing messages such as ‘IDoc created’, ‘The recipient exists’, ‘IDoc was successfully passed to the port’, ‘Could not book the invoice because…’
Different IDoc types are available to handle different types of messages. For example, the IDoc format ORDERS01 may be used for both purchase orders and order confirmations.
tSAPIDocInput and tSAPIDocOutput are Talend’s components to use ALE / IDoc.
BAPIs can also be called asynchronously via ALE. All new IDocs are even based on BAPIs.
SOAP Web Services
SAP supports SOAP web services. Not just SAP AS Java, but also SAP AS ABAP! Integration can be realized with Talend’s ESB / Web Service components such as tESBRequest, tESBResponse, or tESBConsumer.
Installation of SAP Server and Client
Installation can take about 6 to 8 hours, but it is an “all in one installation”, i.e. you can install it overnight. Steps for installation:
- Get yourself a Windows 7 64 Bit Laptop or VM with 8+ GB Ram and 50+GB free disc space
- Get a SAP community account (for free, just register): http://scn.sap.com/welcome
- Download SAP NetWeaver (Software Downloads –> SAP NetWeaver Main Releases: http://www.sdn.sap.com/irj/scn/nw-downloads
Download current version of SAP NetWeaver Application Server ABAP 64-bit Trial
- Install SAP Server: Follow Installation guide – a HTML website included in the download in root of extracted download folder (start.htm –> there click on “Installation” link)
- Install SAP Gui (Rich Client Frontend): start.htm –> there click on “Install SAP GUI” link and follow instructions
- Download the SAP JCo for the operating system on which your connector is running. The SAP JCo is available for download from SAP’s website at http://service.sap.com/connectors. You must have an SAPNet account to access the SAP JCo (if you do not already have one, contact your local SAP Basis administrator).
Usage of SAP Server
- Hint: You have to use a Windows user which has a password (as you need to enter Windows credentials when stopping SAP). If you have a Windows user without a password (for instance if you use Windows within a VM on your Mac), SAP cannot process these credentials (i.e. it cannot process an empty password field) –> Change your Windows password before starting SAP
- Start the management console (Windows Startmenu –> Programs –> SAP Management Console)
- Start and Stop the SAP server (Right Click on “NSP” –> Start / Stop)
- Default User: SAP* (SAP system super user)
- Password: The one which you entered at installation of SAP Netweaver, e.g. admin123
Usage of SAP Client
A SAP client should be used to get information about the SAP system (functions, data, etc.) similarly to using e.g. MySQL Workbench to get information from a MySQL database. SAP GUI (view layer) communicates with SAP AS ABAP (business logic layer). The application server communicates with the relational database (db layer).
Different clients are available for SAP:
- SAP GUI Windows
- SAP GUI Java
- Web Browser
- External RFC-Program
For local development demos, SAP GUI Windows is probably the best alternative. Start SAP GUI Windows by:
- Clicking shortcut “Windows Start menu –> SAP Frontend –> SAP Logon”
- Entering username and password
- Clicking logon
In SAP, you call SAP programs via SAP transaction codes. Important transactions codes are for example:
- BAPI: BAPI Explorer, view all sap BAPI’s
- SE16: Data browser, view/add table data
- SE38: Program Editor
Here is a list of several other important transaction codes: http://www.sapdev.co.uk/tcodes/tcodes.htm
Installation of Demo Data
The SAP installation includes some demo data. As most people do not want to install “real” SAP modules such as SAP FI, SAP CRM or SAP BI on their local system, this demo data is perfect for demos using Talend’s SAP connectors.
To install the flight demo on a local SAP system, you just have to open the ABAP Editor (Transaction: SE38) and execute the program SAPBC_DATA_GENERATOR. This program generates example data within the flight tables and does some further initializations.
Here is a good tutorial with more information and how to test the flight application: http://help.sap.com/saphelp_erp60_sp/helpdata/de/db/7c623cf568896be10000000a11405a/content.htm
Configuration of Talend Studio to use SAP Components
Talend’s SAP components are already included in the Studio. However, two further steps are required to be able to use them:
- Copy sapjco3.dll to the directory c:/Windows/System32
- SAP Java Connector jar must be added
- Copy sapjco3.jar to the directory “Talend/Studio/lib/java”
- (Re-) Start Talend Studio
- Check if SAP library is added successfully
- Open view “Talend modules” (Eclipse –> Windows –> Show View –> Talend –> modules)
- Sort by column “Context”
- Look for “tSAP*” contexts and check if sapjco3.jar has status “installed”
Usage of SAP Components with Talend Studio
This section describes how to use Talend’s SAP components and the SAP Wizard in general (using one specific example for calling a BAPI). Detailed descriptions of all SAP components (for using BAPIs, RFCs, IDocs, BW, etc.) are available in the documentation Talend_Components_RG_X.Y.Z.pdf at www.help.talend.com.
Connection to a SAP System
A connection to a SAP system can be done “built-in” or via “Metadata –> SAP Connections” (the latter only in enterprise version). Using the latter has several advantages:
- Reuse connection configuration
- Quick check if connection to SAP works
- Wizards for retrieving functions from SAP (instead of handwriting without Wizard)
- Quick test with test parameters if function works before finishing
Development Lifecycle for a SAP Job
Development lifecycle for SAP job:
- Create connection (if not existing yet)
- Right click on Metadata –> SAP Connection
- Create SAP Connection
- Follow wizard
SAP JCO Version: 3
Password: “admin123” –> as you defined it while installation
System number: “00”
- Retrieve function (BAPI / RFC)
- Right click on created connection
- Click on “Retrieve SAP Function”
- Enter search filter (e.g. BAPI_FL*)
- Click on “Search”
- Select and double click on your function (e.g BAPI_FLCUST_GETLIST)
- You see all input, output and table parameters for this SAP function
- Click on “Test in” –> Here you see parameters in more detail:
- You now have to define which input and output parameters you want to use –> remove all other by selecting them and clicking “remove” button
- Hint: If you do not remove an input parameter, you usually have to enter a value for it!
- Select the output type – can be a single (single record), a table (list of records), or a structure output
- Hint: Difference between table and structure in SAP: http://www.sapfans.com/forums/viewtopic.php?f=12&t=119794
- If you want to do a quick test: Enter values for input parameters (if there are any for your function call), then click “launch” button
- In this example, there is only an optional input parameter MAX_ROWS
- you should see data in the output fields
- In this example, you see the record with CUSTNAME “SAP AG” and STREET “Neurottstr. 16”
- Click “finish” button
- Under “Metadata –> SAP Connections –> “Your Connection” –> SAP Functions: There you can now see your function (in this example: BAPI_FLCUST_GETLIST)
- Create SAP Job
- Drag&Drop the created function into a job (without the wizard, you also can enter all data by hand)
- tSAPInput component is proposed automatically. Click ok to add it to your job
- Go to “Initialize Input” and add parameter values
- In this example, there is just the parameter “MAX_ROWS”
- Hint: The parameter value can be changed from a hardcoded value to a variable, of course (just click CONTROL SPACE on your keyboard to get access to all available variables via code completion in your Studio)
- Go to the tSAPInput component and add the desired output mapping (i.e. which values you want to process further with other components
- Scroll to the bottom to “Outputs”
- Add the correct table / structure name (in this example: “CUSTOMER_LIST”)
- Click on Mapping (which is empty and has to be filled)
- Click on “Mapping”, then click on “…”
- Add the wanted output columns of your SAP function
- Add the same names at the column “Schema XPathQueries” (do not forget the double quotes here!)
- Click “OK” button
- Connect the tSAPInput component to a tLogRowComponent and synchronize the schema
- Hint: Always try out if this works before adding further logic to your job!
- Run and test your Job (you will see five rows logged (as you have configured MAX_ROWS = 5
That’s it. Now enjoy Talend’s SAP components 🙂
Kai Wähner (Twitter: @KaiWaehner)
Thank you very much for your post.I´m following its steps to connect to a BAPI to retrieve records from ISU tables, everything is OK with few results but when the BAPI retrieves a large number of records, the BAPI returns an exception: time limit exceeded.
I need to execute the BAPI in background processing way. How can I perform this way with talend components? or how I can configure the timeout with talend components?
Thanks in advance
AFAIK, SAP BAPI objects do not offer configuration for latency. There is also no background processing for BAPI calls. BAPI calls are batch processes. Your exception sounds like a SAP problem, not a Talend problem.
If I am wrong, and it is a Talend problem, it sounds like a bug to me. Feel free to create a JIRA ticket (as it is open source) 🙂
great post and approach, but I feel some hacking and workaround here (sapgui, custom parametered ABAP scripts, etc). And that’s not what serious data rockstars deserve 🙂
There is a much easier and more effective way to wire your favorite ERP.
Besides you won’t be stuck with the RFC limitations (row limits, fetching only one table, etc) you can also achieve faster loads. Also this method is still secure, auditable and it is much easier to handle metadata.
Long story short, here’s our solution: http://plsapconnector.com
It would be really interesting if you’d give this one a shot too – and give us some honest feedbacks.
Thanks in advance,
Our partner tells us that they expect XML IDocs files being sent to Talend.
How can I send XML IDocs to Talend ?
Through transaction WE21, we can define XML port but how shall we RFC-trigger Talend !?
While discussing with Talend, SAP understands he has to send XML structure.
However, it only send the SDATA cluster field of data records whereas it should send all data records fields !
When using the XML port (WE21), it works fine.
Do you know why and how to force to convey the whole structure and fields with RFC ?
Parsing every sdata would be a heavy task on TALEND and we don’t know how !
sorry, I do not know more details. Talend’s SAP components only support basic integration problems at the moment (more features are on the roadmap). Today, the solution is not as sophisticated as connectors from IBM, TIBCO, etc.
You should discuss your problem with Talend presales or product management to see if your use case can be solved with Talend.
Thanks for the quick reply. The concerned customer opened a ticket with Talend, let’s hope we get new ideas.
tSAPInput can launch a chosen SAP function module, it may work with more flexibility compared to tSAPIDocinput !?
Thank you for your work,
I have a problem with BAPI_ACC_DOCUMENT_POST function when I use tSAPInput.
I have to insert a stock valuation to an SAP system, I use BAPI_ACC_DOCUMENT_POST function, composed structures, for my case the function is divided into 04 structures:
1) DOCUMENTHEADER: contains company name (BP01), USERNAME,HEADER,…
2) ACCOUNTGL: Contains all brands, general accounts, amounts, … of the company named in the structure DOCUMENTHEADER.
3) CURRENCYAMOUNT: contains amounts, currency and a counter on the company named in the structure DOCUMENTHEADER.
4) CRITERIA: contains information such as brand … for each line of the named company.
I have to recover for each company all its lines, so it must be dynamic and so far, I only found examples where data is manually injected into the BAPI_ACC_DOCUMENT_POST function.
Can you give me an idea of how to proceed?
Thank you in advance.