You can do this couple of ways.

1. By checking depth of the category/sub-category. If we are browsing main/parent categories then show a custom template, if we are on sub-categories then show another template and this goes on.
2. We can check if a category is ending category of not. Ending category means if a category does not have any child then it should show default category.php template.

Which method you should use is depends on your project requirements. Here I am going to explain first one because it’s very flexible and I think will suit your purpose.

WordPress have template_include function that executes immediately before WordPress includes the predetermined template file. This can be used to override WordPress’s default template behavior.

<span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">wpse_template_check</span>(<span class="hljs-params"> <span class="hljs-variable">$template</span> </span>) </span>{
    <span class="hljs-keyword">if</span> ( <span class="hljs-title function_ invoke__">is_category</span>() ) {
        <span class="hljs-comment">// Get category information</span>
        <span class="hljs-variable">$cat</span> = <span class="hljs-title function_ invoke__">get_query_var</span>( <span class="hljs-string">'cat'</span> );
        <span class="hljs-variable">$cats_str</span> = <span class="hljs-title function_ invoke__">get_category_parents</span>( <span class="hljs-variable">$cat</span>, <span class="hljs-literal">false</span>, <span class="hljs-string">'%#%'</span> );
        <span class="hljs-variable">$cats_array</span> = <span class="hljs-title function_ invoke__">explode</span>(<span class="hljs-string">'%#%'</span>, <span class="hljs-variable">$cats_str</span>);
        <span class="hljs-variable">$cat_depth</span> = <span class="hljs-title function_ invoke__">sizeof</span>( <span class="hljs-variable">$cats_array</span> )-<span class="hljs-number">2</span>;

        <span class="hljs-comment">// Check category depth</span>
        <span class="hljs-variable">$new_template</span> = <span class="hljs-title function_ invoke__">locate_template</span>( <span class="hljs-keyword">array</span>( <span class="hljs-string">'category-custom-template.php'</span> ) );
        <span class="hljs-keyword">if</span> ( <span class="hljs-variable">$cat_depth</span> == <span class="hljs-number">0</span> && <span class="hljs-string">''</span> != <span class="hljs-variable">$new_template</span> ) {
            <span class="hljs-keyword">return</span> <span class="hljs-variable">$new_template</span>;
        }
    }
    <span class="hljs-keyword">return</span> <span class="hljs-variable">$template</span>;
}
<span class="hljs-title function_ invoke__">add_filter</span>( <span class="hljs-string">'template_include'</span>, <span class="hljs-string">'wpse_template_check'</span> );

This function calculates category depth and checks if current category is main/parent categories. If it’s true and category template exists then it will replace category template to category-custom-template.php.

I have tested it and it’s working but you should note that it’s based on category depth so all parent categories will have same custom template.

Although you can customize this function to do more and as fit your requirements.