Webhook

Use webhooks to get notified about events related to the SERPHouse Schedule API Such as Google Trend Schedule Search and Schedule SERP.

Webhooks (Web Callback, HTTP Push API or Reverse API) are one way that a web application can send information to another application in real-time when a specific event happens. You can configure your webhook from Webhook Setting Page

Pingback URL

Pingback is also known as callback, It alert you when desired task is completed.

Pingback URL will alert you when your scheduled search have been complete, For e.g if you have sent 100 keywords batch request and now once SERPHouse server completes those keyword processing they will start hitting pingback_url if you have provided URL.

Once you receive pingback request from our server that means your keyword search has been completed and now it’s ready to retrive data from our server using GET SERP Result API.

Our Pingback Feature will served via HTTP GET Request.

Postback URL

Postback is commonly known as a Webhook. While using our Batch API (Delayed SERP API) postback url helps you to reduce your backend work.

While sending Batch Processing request you are allowed to send upto 100 keywords in single request with postback_url to each keywords you sent.

Once your request come to SERPHouse server, Our Backend makes a Queue of your keywords and executes on concurrent thread. As soon as our backend completes keyword processing, You will get an HTTP POST Request on your provided postback_url.

As soon as you receive our request, You have to safely validate and store data on your storage and free up our request within a specified timeout.

Our Postback Feature will served via HTTP POST Request with JSON Body.

Webhook SSL verification?

Webhook SSL verification refers to the process of verifying the authenticity and validity of an SSL certificate used by a webhook endpoint. When a webhook is established between two systems, it is crucial to ensure secure communication and prevent potential security risks. SSL (Secure Sockets Layer) certificates are used to encrypt data transmitted between the systems, providing a secure connection. SSL verification involves checking the SSL certificate presented by the webhook endpoint to ensure it is issued by a trusted certificate authority (CA) and that it has not expired or been revoked. The verification process confirms the webhook endpoint's identity and ensures that the communication is protected against potential man-in-the-middle attacks or unauthorized access. To perform SSL verification, the system initiating the webhook request typically checks the SSL certificate chain, including the root CA and intermediate certificates, to ensure they are valid. It verifies that the common name or subject alternative name (SAN) in the certificate matches the endpoint's domain name. Additionally, the system checks if the certificate is within its validity period and hasn't been tampered with or revoked. If the SSL verification fails, it indicates a potential security risk, and the webhook request may be rejected or considered untrustworthy. Implementing SSL verification for webhooks helps ensure the integrity and security of data exchanged between systems, protecting against unauthorized access and data breaches.

You can enable or disable SSL Verification for your webhook from here

Webhook Retry Strategy

Webhook retry strategy allows our Webhook server to make multiple delivery attempts when your server is temporarily unavailable or unreachable. Here’s how it works:

1. First Attempt: As soon as the results are ready, Our webhook server will try to deliver the data immediately.

2. Second Attempt: If the first delivery attempt fails, Webhook Server will retry after 100 seconds.

3. Third Attempt: A third retry will be made 1000 seconds after the second attempt.

4. Fourth Attempt: A final retry will occur 1800 seconds after the third attempt.

After these four attempts, if the webhook delivery still fails, it will be marked as a permanent failure. At this point, you can manually retrieve your results using our SERP GET API

Why This?

Retry strategy provides several benefits:

• Enhanced Reliability: Multiple retries increase the chances of successful delivery, even if temporary issues prevent the first attempts.

• Improved Control: If all retries fail, you will have the option to manually retrieve the data via the SERP GET API, ensuring that no result is ever lost.

• Optimized User Experience: By spacing out the retry attempts, we reduce the risk of overwhelming your system or server with frequent retries.

How to prevent Duplicate Webhook

There could be scenarios where your endpoint might receive the same webhook event multiple times. This is an expected behaviour based on the webhook design. In such a scenario, we recommend you to follow Idempotency.

You could be receiving the same events multiple times as SERPHouse follows at-least-once delivery semantics. In this approach, if we do not receive a successful response from your server, we resend the webhook. There could be situations where your server accepts the event but fails to respond in 5 seconds. In such cases, the session is marked timeout. It is assumed that the webhook was not processed and is sent again.

Handy Tips

To prevent an event from being missed:

• Ensure you configure your server to handle or receive the same event details multiple times.

• Check the value of the x-serphouse-task-id in the webhook request header. The value for this header is unique per event and can help you determine the duplicity of a webhook event.

Validate Webhooks call from SERPHouse

Validate the webhook before you start using them.

When your webhook secret key is set, SERPHouse uses it to create a hash signature with each payload. This hash signature is passed with each request under the x-serphouse-signature header that you need to validate at your end.

Handy Tips

If you have changed your webhook secret, remember to use the old secret for webhook signature validation while retrying older requests. Using the new secret will lead to a signature mismatch.

Header	Description
x-serphouse-signature	The hash signature is calculated using HMAC with SHA256 algorithm; with your webhook secret set as the key and the webhook request body as the message.

You can also validate the webhook signature yourself using a HMAC as shown below:

HMAC Hex Digest

key                = webhook_secret
message            = webhook_body // raw webhook request body
received_signature = webhook_signature

expected_signature = hmac('sha256', message, key)

if expected_signature != received_signature
	throw SecurityError
end

Example Verify Signature Code :

NODE.JS

var http = require("http");
const crypto = require('crypto');
const express = require('express')
const app = express()
var bodyParser = require('body-parser')
const sigHeaderName = 'x-serphouse-signature';
const sigHashAlg = 'sha256';
const sigPrefix = ''; //set this to your signature prefix if any
const secret = "yoursecretkey"; //your webhook secret key from your webhook settings https://app.serphouse.com/wehook-setting
app.use(bodyParser.json(
    {
        verify: (req, res, buf, encoding) => {
            if (buf && buf.length) {
            req.rawBody = buf.toString(encoding || 'utf8');
            }
        },
    }
));
function validatePayload(req, res, next) {
    if(req.get(sigHeaderName)){
        //Extract Signature header
        const sig = Buffer.from(req.get(sigHeaderName) || '', 'utf8')
        //Calculate HMAC
        const hmac = crypto.createHmac(sigHashAlg, secret)
        const digest = Buffer.from(sigPrefix + hmac.update(req.rawBody).digest('hex'), 'utf8');
        //Compare HMACs
        if (sig.length !== digest.length || !crypto.timingSafeEqual(digest, sig)) {
            console.log("Unable to verify signature");
            return res.status(401).send({
                message: `Request body digest (${digest}) did not match ${sigHeaderName} (${sig})`
            });
        }
        else {
            console.log("Signature successfully verified ");
        }
    }
    return next()
}
app.use(validatePayload);
app.post('/webhook', function(req, res) {
    res.send("All is well, I am here after the successful verification")
})
http.createServer(app).listen(8081)
console.log('Server running at http://127.0.0.1:8081/');

PHP

$secret_sign_key = 'yoursecretkey'; //your webhook secret key from your webhook settings https://app.serphouse.com/wehook-setting
$webhook_content = file_get_contents('php://input'); //raw webhook request body
$received_signature = $_SERVER["HTTP_X_SERPHOUSE_SIGNATURE"];
$expected_signature = hash_hmac('sha256', $webhook_content, $secret_sign_key);
if($expected_signature != $received_signature) {
    echo "Unable to verify signature";
} else {
    echo "Signature successfully verified";
}   

PYTHON

secret_sign_key = 'yoursecretkey' #your webhook secret key from your webhook settings https://app.serphouse.com/wehook-setting
webhook_content = request.data #raw webhook request body
received_signature = request.headers['x-serphouse-signature']
expected_signature = hmac.new(secret_sign_key.encode(), webhook_content, hashlib.sha256).hexdigest()
if(expected_signature != received_signature) :
    print("Unable to verify signature")
else :
    print("Signature successfully verified")

RUBY

require 'sinatra'

post '/posts' do
    secret_sign_key = "yoursecretkey" #your webhook secret key from your webhook settings https://app.serphouse.com/wehook-setting
    webhook_content = request.body.read #raw webhook request body
    received_signature = request.env['HTTP_X_SERPHOUSE_SIGNATURE']
    expected_signature = OpenSSL::HMAC.hexdigest(OpenSSL::Digest.new('sha256'), secret_sign_key, webhook_content)
    if expected_signature != received_signature
        puts "Unable to verify signature"
    else
        puts "Signature successfully verified"
    end
end

JAVA

String secret_sign_key = "yoursecretkey" //your webhook secret key from your webhook settings https://app.serphouse.com/wehook-setting
String webhook_content = new String(exchange.getRequestBody().readAllBytes()); //raw webhook request body
Headers requestHeaders = exchange.getRequestHeaders();
List<String> received_signature = requestHeaders.get("x-serphouse-signature");
try {
    Mac hmac = Mac.getInstance("HmacSHA256");
    SecretKeySpec secretKeySpec = new SecretKeySpec(secret_sign_key.getBytes(StandardCharsets.UTF_8), "HmacSHA256");
    hmac.init(secretKeySpec);
    byte[] calculatedSignatureBytes = hmac.doFinal(webhook_content.getBytes(StandardCharsets.UTF_8));
    String expected_signature = Hex.encodeHexString(calculatedSignatureBytes);
    if(expected_signature.equals(received_signature.get(0)))  {
        System.out.println("Signature successfully verified");
    } else {
        System.out.println("Unable to verify signature");
    }
} catch (NoSuchAlgorithmException | InvalidKeyException e) {
    e.printStackTrace();
    System.out.println("Exception found");
}

C#/.NET

string secret_sign_key = "yoursecretkey" //your webhook secret key from your webhook settings https://app.serphouse.com/wehook-setting
string webhook_content = requestBody; //raw webhook request body
string received_signature = request.Headers["x-serphouse-signature"];

byte[] keyBytes = Encoding.UTF8.GetBytes(secret_sign_key);
byte[] messageBytes = Encoding.UTF8.GetBytes(webhook_content);

byte[] computedSignature = new HMACSHA256(keyBytes).ComputeHash(messageBytes);
string expected_signature = String.Concat(Array.ConvertAll(computedSignature, x => x.ToString("x2")));

if (expected_signature.Equals(received_signature)) {
    Console.WriteLine("Signature successfully verified");
} else {
    Console.WriteLine("Unable to verify signature");
}

GO

secret_sign_key := "yoursecretkey" //your webhook secret key from your webhook settings https://app.serphouse.com/wehook-setting
webhook_content, err := ioutil.ReadAll(r.Body) //raw webhook request body
if err != nil {
	http.Error(w, "Failed to read request body", http.StatusBadRequest)
	return
}
received_signature := r.Header.Get("x-serphouse-signature")
h := hmac.New(sha256.New, []byte(secret_sign_key))
h.Write(webhook_content)
expected_signature := hex.EncodeToString(h.Sum(nil))

if hmac.Equal([]byte(received_signature), []byte(expected_signature)) {
	fmt.Println("Signature successfully verified")
} else {
	fmt.Println("Unable to verify signature")
}

Idempotency

There could be scenarios where your endpoint might receive the same webhook event multiple times. This is an expected behaviour based on the webhook design.

To handle duplicate webhook events:

• You can identify the duplicate webhooks using the x-serphouse-task-id header. The value for this header is unique per event.

• Check the value of x-serphouse-task-id in the webhook request header.

• Verify if an event with the same header is processed at your end.