What if I say, you don’t need to code a single line to perform enterprise-level Integration. Yes !! You heard right. Salesforce is all about low coding, but still, we as a developer are writing codes to perform different types of integration. Let remember my first blog where I come across the SKYVVA Data Loader and have worked with it as the replacement of the Salesforce Data Loader. SKYVVA provide not only the easy data loading and importing as we know with the data loader. SKYVVA provide more to do a complex and enterprise-class integration, which is hard to do with the API approach only. It is a very powerful tool which provides Integration with various advanced features and with no coding. This secret tool can change your organization’s Integration process completely.
The Salesforce standard API
So let’s first understand what are the standard ways to do Integration in Salesforce. I will shortly outline the different types of API’s which we can use for Integration and manipulating your Salesforce organization data. For the completeness of the API documentation and description please refer to Salesforce official help site. Let’s have a look on different API’s:
REST API
REST API is a simple and powerful web service based on RESTful principles. It exposes all sorts of Salesforce functionality via REST resources and HTTP methods. For example, you can create, read, update, and delete (CRUD) records, search or query your data, retrieve object metadata, and access information about limits in your Org. REST API supports both XML and JSON. Because REST API has a lightweight request and response framework and is easy to use, it’s great for writing mobile and web apps.
SOAP API
SOAP API is a robust and powerful web service based on the industry-standard protocol of the same name. It uses a Web Services Description Language (WSDL) file to rigorously define the parameters for accessing data through the API. SOAP API supports XML only. Most of the SOAP API functionality is also available through REST API. It just depends on which standard better meets your needs. Because SOAP API uses the WSDL file as a formal contract between the API and consumer, it’s great for writing server-to-server integrations.
BULK API
Bulk API is a specialized RESTful API for loading and querying lots of data at once. By lots, we mean 50,000 records or more. Bulk API is asynchronous, meaning that you can submit a request and come back later for the results. This approach is the preferred one when dealing with large amounts of data. There are two versions of the Bulk API (1.0 and 2.0). Both versions handle large amounts of data, but we use Bulk API 2.0 in this module because it’s a bit easier to use. Bulk API is great for performing tasks that involve lots of records, such as loading data into your org for the first time.
Streaming API
Streaming API is a specialized API for setting up notifications that trigger when changes are made to your data. It uses a publish-subscribe, or pub/sub, model in which users can subscribe to channels that broadcast certain types of data changes. The pub/sub model reduces the number of API requests by eliminating the need for polling. Streaming API is great for writing apps that would otherwise need to frequently poll for changes.
API Limits
Total limits vary by org edition, license type, and expansion packs that you purchase. For example, an Enterprise Edition org gets 1,000 calls per Salesforce license and 200 calls per Salesforce Light App license. With the Unlimited Apps Pack, that same Enterprise Edition org gets an extra 4,000 calls. Total limits are also subject to minimums and maximums based on the org edition, but we won’t get into that here. If you want to know more, check out the API Request Limits link in the Resources section.
Which API to use in which case?
Choosing the right API for your integration needs is an important decision. Here’s some information on our most commonly used APIs, including supported protocols, data formats, communication paradigms, and use cases. Treat this section as a reference you can return to when you’re considering which API to use.
When to Use REST API
REST API provides a powerful, convenient, and simple REST-based web services interface for interacting with Salesforce. Its advantages include ease of integration and development, and it’s an excellent choice of technology for use with mobile applications and web projects. For certain projects, you may want to use REST API with other Salesforce REST APIs. To build UI for creating, reading, updating, and deleting records, including building UI for list views, actions, and dependent picklists use User Interface API. To build UI for Chatter, communities, or recommendations, use Chatter REST API. If you have many records to process, consider using Bulk API, which is based on REST principles and optimized for large sets of data.
When to Use SOAP API
SOAP API provides a powerful, convenient, and simple SOAP-based web services interface for interacting with Salesforce. You can use the SOAP API to create, retrieve, update, or delete records. You can also use SOAP API to perform searches and much more. Use SOAP API in any language that supports web services.
For example, you can use SOAP API to integrate Salesforce with your org’s ERP and finance systems. You can also deliver real-time sales and support information to company portals and populate critical business systems with customer information. Note that SOAP API has a well-defined standard by W3C consortium and reach a long maturity and thus old application systems still only support soap. Old applications which deliver value to your new digital scenario are still a great asset for your company and thus need to integration to the new digital process.
When to Use Bulk API
Bulk API is based on REST principles and is optimized for loading or deleting large sets of data. You can use it to query, queryAll, insert, update, upsert, or delete many records asynchronously by submitting batches. Salesforce processes batches in the background.
SOAP API, in contrast, is optimized for real-time client applications that update a few records at a time. You can use SOAP API for processing many records, but when the data sets contain hundreds of thousands of records, SOAP API is less practical. Bulk API is designed to make it simple to process data from a few thousand to millions of records.
The easiest way to use Bulk API is to enable it for processing records in Data Loader using CSV files. Using Data Loader avoids the need to write your own client application.
When to Use Streaming API
Use Streaming API to receive near-real-time streams of data that are based on changes in Salesforce records or custom payloads. For Salesforce record changes, Salesforce publishes notifications when the changes occur. For custom notifications, you can publish event messages. Subscribers can receive notifications using CometD – an implementation of the Bayeux protocol that simulates push technology. Clients can also subscribe to some types of events with Apex triggers or declaratively with Process Builder and Flow Builder. Use the type of streaming event that suits your needs.
Push Topic Event
Receive changes to Salesforce records based on a SOQL query that you define. The notifications include only the fields that you specify in the SOQL query.
Change Data Capture Event
Receive changes to Salesforce records with all changed fields. Change Data Capture supports more standard objects than Push Topic events and provides more features, such as header fields that contain information about the change.
Platform Event
Publish and receive custom payloads with a predefined schema. The data can be anything you define, including business data, such as order information. Specify the data to send by defining a platform event. Subscribe to a platform event channel to receive notifications.
Generic Event
Publish and receive arbitrary payloads without a defined schema.
Nature of API and its principal use case
As you have seen the different available API’s for integration and it’s use cases now let us look deeper into the nature and the principle behind the API. API become popular in the area of moving the application to a distributed network where business logic does not happen as the whole bunch of logic inside a monolith anymore. The decomposition and the distribution become the design principle of creating a new area of application which can be easily used over a distributed network such as the internet. To be able to communicate with now autonomy piece of software and application which can be specialized on a business domain the API becomes the vehicle to do that. The software application has to communicate with each other using API.
The application was decomposed to logical units which have to communicate with each other to fulfill the business task. Most of the time this communication is based on a synchronous way thus happen in real-time. This kind of communication is good to provide interaction between the application and the business user. On the other side, doing integration happened mostly in the background and run autonomously without having a user sitting and waiting for any response of an API. The work has to be done in the background and some time with a high volume of data automatically. With other words, the communication has to happen in an asynchronous way and need to support background and batch processing of high volume data.
The integration pattern
Looking to the available Salesforce standard API’s above except the bulk api you can see that they are operating in the synchronous pattern and thus follow the pattern of ‘tight coupling’ integration. When you are dealing with integration you will come across two principal approaches of doing the integration.
- Tight Coupling
- Loose Coupling
The following picture shows the two pattern:
A tightly coupled integration is one that creates a dependent relationship between Salesforce and whatever systems it’s being linked to by spreading the integration of business logic across those systems. While a tightly coupled integration could function just fine, it will inevitably cause scalability and maintenance problems in the long term. A loosely coupled integration, on the other hand, keeps the integration business logic separate from each system, thus create independence, interoperability, and decoupled application. This allows you to more easily modify your integration logic and to make changes to each individual system without fear of breaking any existing functionality.
Tight Coupling
With the Salesforce soap and rest API, you can greatly do simple so-called CRUD operation with the data which operate synchronously. This means that the caller application or client need to wait for the API to be finished. What happened with the client when the API run into an endless loop because of having a recursive trigger on sObject. Are the clients still responsive? Can the user use the application or is it going to be frozen? This kind of problems occurs when using tight coupling pattern with the wrong design. Tight coupling is a valid and great pattern to use and not saying that this pattern is useless. The things are that in integration world the use case is more and better fit to the so-called loose coupling of application inter-communication.
Due to the decomposed designed of having separate functional units in the modern software design such a tight coupling cause too many dependencies. It is such a big and monolith block which cannot easily be distributed over the internet. Lightway and functional encapsulated app is now the future and thus need to decouple the communication between that app. With this evolution in the last few decades, we could see new application vendors entering the market with new application software which provides a specific and well-suited business package. Therefore it is a strong demand to integrate them together in the right way with the right architecture.
Here some thought and fact to consider when you plan to use the synchronous and tight coupling architecture for your integration project.
- Tight coupling model provides more interdependency on both the Integration systems.
- It required more coordination between two systems while Integration as both are equally co-ordinating with each other during Integration data processing in order to success.
- The caller needs to wait till API is finished which will result in blocking of the sender who is ending the requested data.
- This waiting for API callouts results in bad user experience and bad performance for the client. Sometimes it appears to your users that your application is frozen. This will cause dislike and business damage if your user is not using your app anymore.
- The client cannot use all its resources due because of waiting and idle time which results in frustration.
- This API always follow Ping-Pong pattern to synchronize API between caller and provider hence more efforts to be taken when its tightly coupled Integration.
On the other side using tight coupling is also a valid way to do integration when your business cases need real-time interaction to support your business users. Only in this case, the disadvantages described above can be ignored.
Loose Coupling
As you see nowadays applications are breaking from its monolith structure of small, smart and maintenance able unit there is not tight coupling in the nature of application anymore. Thus the tight coupling pattern doesn’t fit much and application need to talks and interacts loosely together without having a user or process waiting for others. This kind of integration causes less inter-dependency and are thus great to maintain and change flexibility to meet any kind of new business requirements.
Loose coupling follows the principle of the asynchronous communication pattern. Imagine calling somebody by phone where you always need somebody to be available at the time you start the communication. With the asynchronous pattern, you don’t need your communication partner to be available and thus create less dependency on communication. This is for example when you write an email to a recipient. You are not blocked for hours you just sent your message and can turn around to do other things. This is one of the main advantages of not being blocked by the communication partner.
Lose coupling solve the disadvantages you have with the tight coupling pattern described above. It doesn’t need and bound huge resources on both side e.g. sender and receiver for doing the communication. It releases the client e.g. caller quicker and thus doesn’t create the impression to the user that the application freeze. Furthermore, if you integration scenario involved exchanging mass data e.g., for example, updating of millions of the product prices from an ERP system then there is no way to use the asynchronous communication to be able to send such amount of data using the bulk API.
There are a lot of great books and resources available over the internet which you can read to more precisely understand the communication pattern using for integration domain. In this blog, we cannot cover and explain all the theoretical aspect of the communication pattern because it would exploit the size of this blog. Let see in the following chapter how I found such a solution with SKYVVA which support both e.g. tight and loose coupling.
Integration using the API only approach
Now as we have seen and understood the two existing approach and pattern for doing integration let have a deeper look at how we can do the integration to manually program against the Salesforce standard API. Salesforce now provides both flavors of integration API which supports SOAP and REST. For a few years ago you can only use the SOAP API to do the integration and connect to your Org. Therefore older application still existing which uses the SOAP API. The picture below shows the view when doing integration using synchronous pattern with the Salesforce standard SOAP/REST API.
It shows the use case with an SAP cloud ERP application as a client which use API to synchronized data like Account, Contact, sales order, invoice, etc. In such a business case you have to deal with mass data to synchronize between SAP and Salesforce application system. It required autonomous data synchronization between that two application system without to have any users involved and it needs batch processing because of the mass of data.
The characteristic of using the Salesforce standard API is that it operates synchronously meaning that the client application has to wait until the operation which is triggered by the API is done. This can cause all the disadvantages we have seen above with the usage of the tight coupling approach. The most seen issue with using the standard API is that the client application seems to be frozen when there are too many requests at the same time.
Another aspect of using the Salesforce standard API is that it provides a very simple CRUD operation capability. But in real integration scenario, we need more than only create (C), read (R), update (U) and delete (D) the business record as for example a quote. We need more complex logic to do first before we are doing those basic REST operation. For example, before updating the quote we need to check based on the sales area assignment in SAP if this quote belongs to the correct territory area in Salesforce. So we need some business logic which needs to do beforehand.
The real integration scenario for an enterprise is much more than only doing the simple CRUD-operation. You need a full-fledged of services on the Salesforce side to be able to provide a stable integration not only in term of simple connectivity but also to support the daily operation when something is going wrong. Doing this you need an integration service layer sitting on the Lightning platform such as I could found with the SKYVVA solution.
Looking to the picture above you can see the filling gap (the black whole in the above picture) which a service layer could bring to your integration need. It goes beyond the connectivity and provides you and your Salesforce team an easy way to develop, maintain and operate your daily integration need. This is what the SKYVVA service layer adds to the Salesforce Lightning platform which is missing for having the right and robust tool for doing the integration to and from Salesforce.
Why integration needs both patterns?
As integration problems and use cases are too different we cannot say that the tight or loose coupling will be the only right solution for a requirement. If you need real-time to see, for example, the availability of a product on the stock then you simply need to solve this requirement using the tight coupling pattern and thus provide the real-time experience to your user who is waiting on his mobile device to see. If you need to synchronize the account address data which is enough to see them in the next day then you can use the loose coupling pattern to delay the batch processing of the account data update to a nightly job. Thus you don’t disturb the business user who is working on the main business hour during the day.
One approach and architecture style is not enough in solving today integration need for Salesforce eco-system. If you are using a technology which supports only one of that communication pattern you will be lost in providing a stable, reliable and on-time connected digital business with your Salesforce platform. You simply need a tool which supports you in doing integration using both patterns in an effective way and without demanding from you to be an API developer expert. With SKYVVA I found such a toolset providing both patterns with the native Salesforce technology. No additional tools and middleware are needed to provide enterprise-class integration.
After having explained the Salesforce standard API, the two integration pattern and showing you the need of having both patterns to build any integration now lets us have a look to a few features offered by SKYVVA solution. Note that all the feature we will show you below is possible only because of the service layer from SKYVVA which in fact is the decoupling layer. Let me also repeat here again that SKYVVA supports both patterns and let you do a real-time integration scenario by by-passing the SKYVVA staging layer in the same manner you can do with the Salesforce standard API. The differences are that you get more functionality with the SKYVVA solution then when doing the integration manually yourself or let it done by a developer team who need deep knowledge and skill for API programming.
The SKYVVA added-value we want to showcase in the below chapter are the following:
- Message Monitor, which give you a handy tool for monitoring incoming and outgoing messages
- Message Reprocessing, which allow you to correct wrong incoming data a do reprocess without having to ask the sender application to send the data again.
- Alerting, ease your life not to actively and permanently looking to the monitor to see what is going wrong. Instead, you will get an alert and notification easily to your mobile device or chatter group.
How to monitor in case something is going wrong?
Let us start with a question if your interface and integration you have developed always run smoothly without any issue? Do you have never had a failed data integration due to incorrect data e.g. wrong date and currency format or the user entered simply incorrect data at the sending application? How do you deal with all kind of errors you face at the daily operation. If you are a Salesforce experienced developer you probably have no problem to find out the root cause for integration error. But do you want to spend your resources every day for doing the integration error which can be easily done for example from your admin or end users colleagues? This is the point and time-saving issue I come across while using the SKYVVA solution which provides me all these benefits in a very simple way.
With SKYVVA you now have end-2-end monitoring because the messages are kept for monitoring purpose in the SKYVVA service layer and staging area. Without the SKYVVA service layer, you would have only the monitoring capability provided by the sender application and the middleware. This value-added feature is provided exclusively by the SKYVVA service layer.
The screen below shows an example of the monitoring screen where you have different options to find the messages. For example, the most used searching options are to find the message based on its content e.g. by entering the account name. With one mouse click, you have a clear view of your data and can compare easily on a business level without having to be a Salesforce developer expert to be able to read the debug log and traces.
It provides flexibility to monitor your Integration data by specific date and time as well. Hence you don’t need to be always present in front of your system to perform data Integration.
Depending on the processing different message status filter options are available like
Completed: When API call is successful and data is successfully posted the status is completed.
Failed: When data is posted successfully but due to some reason it is failed for example due to mismatch of data type like its text field in sender but it’s number field in Salesforce, in that case, it cannot post the record. With the monitoring screen, you can actually monitor the reason for the failure of data after the API call.
Pending: When your data is sent by the sender but pending to be posted in salesforce then message status change to pending.
Canceled: When data is canceled after API calls then message status changed to canceled
Please refer to the SKYVVA online tutorial and documentation page to see the detail of the monitoring capability here:
https://apsara-consulting.com/docs/skyvva-administration-guide-v2-41-lightning/message-monitoring/
Easy data fixing and correcting using the reprocessing feature
Imagine an example where you have to bring your sales quote and order from an ERP-System such as SAP to Salesforce for your sales representative to provide a clear picture of customer quote and order amount. Quote and order can be synchronized to Salesforce immediately with the SAP quote which has been changed in SAP. Such a process happened in the back-office by your colleagues who are using SAP to create and change the quote and order. Now imagine that being human people your colleague can make a mistake and enter the wrong data into the quote. Instead of entering the correct currency format he/she has entered a wrong one which is not recognized by Salesforce. The posting of the quote thus fail.
What to do now? You can ask your colleague to correct the data on the SAP system and send it again. What happened when it is close to the business day and your colleagues left the office already? There is no one who can correct the wrong currency. The problem you have now is that it is close to the end of the business day and you have to send the correct quote from Salesforce to the potential client. Of course, you can shift to send the quote tomorrow. But if you are not the only vendor who sends the quote to that potential client and if your competitor can send by the end of the business day and not you what is going to happen? Imagine this scenario which can damage your business!
Such a scenario and use case happened in the reality and you need a possibility to independently do the correction and reprocessing of the application data posting e.g., in this case, to update the Salesforce quote with the correct currency from SAP. SKYVVA allow you to do such a data correction with the ability to reprocess the data immediately. Thus you can correct the wrong currency and post your quote successfully. Now you can send the quote before the end of the day to your potential client. You even can set up a scenario back to send the changed data using the process build process with the SKYVVA callout functionality to update the changes you made back to SAP.
Start to use the message monitor to look for failed messages or jump from your email inbox where you got the alert and jump to the failed message. In the failed message make the necessary data correction and reprocess it.
Why alerting can help you
As you have seen having an additional component to monitor data in Salesforce is a great help for your daily operation to fix integration failure. If you don’t have the monitoring component your only way is to dig into the debug trace and look for possible failure. This is complex, time-consuming and needs experienced apex developer with deep API knowledge. With the monitoring, you can delegate this kind of work to your Salesforce admin and event business users. You are free to develop other nice application your company might need.
Monitoring allows you to actively look, search and see the integration data failure. What to do if you don’t have time to look every 30 minutes to the monitoring? You are mostly busy with meeting and some other task and cannot afford to open for every 30 minutes the monitoring page. This is why alerting comes into place. With the message failure alerting from the SKYVVA solution, you don’t need more to actively care about any data integration failure. Even while you are drinking a cup of coffee you will know when something is going wrong with your data integration. SKYVVA provide you real-time alert and notification to your email inbox, create a task for your user and use the social media functionality using the chatter group to post alert.
With this feature, the SKYVVA solution has added the missing part to the Salesforce platform regarding alerting functionality. Now you have alongside the integration chain alerting capability. It is alerting everywhere and nothing can be get lost anymore. This is possible due to the decoupling and integration service layer of the SKYVVA solution added to the Salesforce Lightning platform.
Summary
Finally, you reach the end of this long blog. There are some more functionality and added value which I come across the SKYVVA integration solution while I have been working with this great native app. For example, regarding the reprocessing feature, you don’t need to do every data reprocessing manually. There are schedulers available in the SKYVVA solution which will do the reprocessing automatically. For data failure like business data record lock due to concurrent processing on the same record by an end user and the integration user at the same time the reprocessing job will resolve it. There is no need to do the reprocessing manually. The scheduler can be flexibly defined e.g. to run for example every 15 minutes or only on Sunday at 10 pm in the evening.
As you have seen it is not the matter or question of if you can do integration or not with the existing Salesforce standard API. The answer is YES you can do everything yourself if you have the right skill and coding experiences. But are you really providing the integration with the same quality and robustness which enterprise-class integration needed if you only use the API connectivity approach? What about supporting decoupling processing between the client and Salesforce? Can this be done using the standard Salesforce API?
What about the error handling including alerting functionality you need for a real robust integration? Do you have all of the required functionality to handle your daily data synchronization operation which includes the monitoring, reprocessing and alerting? If you just need a ping-pong and simple CRUD operation connectivity you can use the Salesforce standard API but if you want to have more or if your business needs a stable and maintenance-able integration and data synchronization then there is no way not to go for a professional solution like a middleware. But what to do when your company doesn’t have the budget to use a full-blown middleware because of cost reason? If you are in such a situation but need an enterprise-class integration solution like a middleware can provide it then definitely my recommendation is to have a look to the SKYVVA app.
Find here the latest release version https://apsara-consulting.com/en/news/ and make the first try with the free SKYVVA Data Loader. For a test version with all the functionality but limited to 30 days of validation fill in the contact formula and ask for a trial key.